.. meta:: :title: Ansible Playbooks for Confluent Platform :keywords: Confluent Ansible, Ansible, Ansible Playbooks :description: This guide provides instructions for using Confluent's Ansible playbooks to automate the deployment of Confluent Platform with different security configurations. .. _cp-ansible-playbooks: Install using Ansible --------------------- Refer to the following sections for information about using `Ansible playbooks `__ provided by Confluent to install the `Confluent Platform `__. .. important:: **Support:** For |cp| 5.3 (and later), the Ansible playbooks are fully supported for those with a Confluent Support contract. Any issues encountered can be reported to Confluent Support at https://support.confluent.io. For those without a Confluent Support contract, issues can be reported using the `cp-ansible GitHub repo `__ and are addressed on a best-effort basis. **Feature recommendations:** For those with a Confluent Support contract, future work and additional features should be filed by opening a case with Confluent Support at https://support.confluent.io. For those without a Confluent Support contract, please review the `Contributing document `_. --------------------------- About the Ansible playbooks --------------------------- Ansible provides a simple way to deploy, manage, and configure |cp|. The `cp-ansible `__ repository provides the playbooks and templates that allow you to easily provision the |cp| in your environment. The Ansible playbooks provided by Confluent perform the following operations: * Installs Confluent Platform packages. * Starts services using systemd scripts. * Provides variables for configuring various security settings between |cp| component services: - SSL (one way and two way) with self-signed certs or custom certs - SASL (Kerberos, Plain) You can install the services listed below using the Ansible playbooks provided by Confluent: * |zk-full| * |ak-tm| * |sr-long| * REST Proxy * |c3| * |kconnect-long| (distributed mode) .. note:: With the exception of |c3|, all |cp| components are installed with Jolokia enabled and metrics exposed. ------------------------- Installation Requirements ------------------------- General Requirements: * Confluent Platform 5.3 or later * Ansible 2.5 or later (on control node) * `Confluent Platform Ansible playbooks `__ * ssh access between Ansible hosts and all other hosts Operating System: * RHEL 7 or later * Ubuntu 16.04 or later * Debian 8 or later Minimum Hardware: * 4 hosts * 8 CPU Cores per host * 32 GB of RAM per host -------------------- Running the playbook -------------------- Using Ansible to install and configure |cp| consists of a few steps. You create an inventory file, set up the variables you need, and then run the playbook. The following steps provide a high-level overview of the procedure. #. Get the source code. :: git clone https://github.com/confluentinc/cp-ansible #. Create a copy of the ``hosts_example.yml`` file. :: cd cp-ansible cp hosts_example.yml hosts.yml #. Edit your inventory file and enter the correct hosts, SSH variables, and custom variables. :: vi hosts.yml #. Confirm Ansible can connect over SSH. :: ansible -i host.yml all -m ping #. Run the ``all.yml`` playbook. :: ansible-playbook -i host.yml all.yml The following sections provide detailed information about generating your inventory file, setting custom variables, and configuring different options for security. ------------------------------ Generating your inventory file ------------------------------ Before running the Ansible playbook you need to generate an inventory file. The inventory file provisions the hosts used for the |cp| components. See `Inventory basics: hosts and groups `__ for more information about the inventory file. An example base inventory file is shown below: :: zookeeper: hosts: ip-192-24-10-207.us-west.compute.internal: ip-192-24-5-30.us-west.compute.internal: ip-192-24-10-0.us-west.compute.internal: kafka_broker: hosts: ip-192-24-10-207.us-west.compute.internal: broker_id: 1 ip-192-24-5-30.us-west.compute.internal: broker_id: 2 ip-192-24-10-0.us-west.compute.internal: broker_id: 3 schema_registry: hosts: ip-192-24-10-207.us-west.compute.internal: control_center: hosts: ip-192-24-12-225.us-west.compute.internal: kafka_connect: hosts: ip-192-24-10-207.us-west.compute.internal: kafka_rest: hosts: ip-192-24-10-207.us-west.compute.internal: ksql: hosts: ip-192-24-5-30.us-west.compute.internal: Save your inventory file onto the host running Ansible as ``hosts.yml``. Enter the following command to verify that Ansible can reach each host: :: ansible -i /path/to/hosts.yml all -m ping This verifies that the Ansible bastion can reach each host over SSH. This is a requirement for running the playbook. If ping fails, you likely need to set up your ``ssh`` variables. See `Connecting to hosts: behavioral inventory parameters `__ for more information about setting up ``ssh`` variables. --------------------------- Setting up custom variables --------------------------- Confluent provides a `sample inventory file `__ with all supported customization variables. Use this file as a reference when setting your configuration properties and when setting up your TLS an SASL configuration variables. You can set custom properties for each component in the inventory file using variables. An example is shown below: :: all: vars: zookeeper: properties: initLimit: 6 syncLimit: 3 kafka_broker: properties: num.io.threads: 15 In the example above, the ``initLimit`` and ``syncLimit`` properties get set in the |zk| properties file. The ``num.io.threads`` property gets set in the |ak| properties file. JMS Exporter ^^^^^^^^^^^^ Additionally, a variable is available that allows you to enable JMX Exporter on your |ak| brokers. The following example shows how to enable this property. :: ## JMX Exporter is disabled by default. When enabled, JMX Exporter jar will be pulled from the Internet and enabled on the broker *only*. ## To enable, uncomment this line: jmxexporter_enabled: true .. _TLS-SASL-configuration-options: -------------------- Configuring Security -------------------- The following sections provide instructions for configuring security. Configuring TLS encryption ^^^^^^^^^^^^^^^^^^^^^^^^^^ By default the components are installed using the PLAINTEXT protocol. This means there is no data encryption. You can enable TLS encryption with one of the following options: * **Self Signed Certs:** A Certificate Authority will be generated by the playbooks and used to sign the certs for each host. * **Custom Certs:** You can provide signed certs and keys for each host as well as the Certificate Authority Cert used to sign the certs. * **Custom Keystores and Truststores:** You can provide keystores and truststores for each host. .. note:: By default the TLS connection is **one-way TLS**. Mutual TLS is also configurable. TLS encryption variables should be added under the *all* group in the hosts.yml file. To turn on TLS encryption set the following parameter to true: :: ssl_enabled: true By default, components are configured with one-way TLS. To turn on TLS mutual authentication, set the following parameter to true: :: ssl_mutual_auth_enabled: true By default, the certs for this configuration are self-signed. To deploy custom certificates, there are two options available: **Custom Certificates** and **Custom Keystores and Truststores**. .. note:: The ``ssl_enabled`` property applies globally to all |cp| components **except** |zk|. Starting from the 5.4 release of the |cp| playbooks, this global setting will apply to |zk| as well. =================== Custom Certificates =================== To provide Custom Certificates for each host, you will need the Certificate Authority certificate, the signed certificates, and keys for each host on the Ansible host. Complete the following steps to use Custom Certificates. #. Set this variable to true. :: ssl_custom_certs: true #. Enter the path to the Certificate Authority Cert used to sign each host certificate. :: ssl_ca_cert_filepath: "/tmp/certs/ca.crt" #. Set the following variables for each host. :: ssl_signed_cert_filepath: "/tmp/certs/{{inventory_hostname}}-signed.crt" ssl_key_filepath: "/tmp/certs/{{inventory_hostname}}-key.pem" The variable ``{{inventory_hostname}}`` in the example shows that Ansible can read the hostnames set in the inventory file. For this reason, you can keep the inventory file shorter if you put the hostname in the filename for each signed certificate and key file. As an alternative, you can set the variables directly under a host: :: schema_registry: hosts: ip-192-24-10-207.us-west.compute.internal: ssl_signed_cert_filepath: "/tmp/certs/192-24-10-207-signed.crt ssl_key_filepath: "/tmp/certs/192-24-10-207-key.pem ================================ Custom Keystores and Truststores ================================ To provide Custom Keystores and Truststores for each host, you will need to have keystores and truststores for each host on the Ansible host, as well as their passwords. Complete the following steps to use Custom Keystores and Truststores. #. Set this variable to true. :: provided_keystore_and_truststore: true #. Set the following variables for each host. :: ssl_keystore_filepath: "/tmp/certs/{{inventory_hostname}}-keystore.jks" ssl_keystore_key_password: mystorepassword ssl_keystore_store_password: mystorepassword ssl_truststore_filepath: "/tmp/certs/truststore.jks" ssl_truststore_password: truststorepass By taking advantage of the ``{{inventory_hostname}}`` variable and setting the same password for each host, you can set these variable once under the *all* group in the ``hosts.yml`` file. As an alternative, you can set these variables under each host as shown in the alternative Custom Certificates example. Configuring SASL Authentication ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Confluent Ansible can configure SASL/PLAIN or SASL/GSSAPI (or Kerberos). By default, there is no SASL authentication. =========================== SASL authentication options =========================== By default the components are installed without any SASL authentication. You can enable the following SASL options: * **PLAIN:** SASL/PLAIN uses a simple username and password for authentication. * **Kerberos:** SASL with Kerberos provides authentication using your Kerberos or Active Directory server. .. note:: Kerberos Key Distribution Center (KDC) and Active Directory KDC configurations are not currently configured by the Ansible playbook. ====================== Configuring SASL/PLAIN ====================== To configure SASL/PLAIN security, set the following option in the *all* group in the ``hosts.yml`` file. :: sasl_protocol: plain ============================== Configuring SASL with Kerberos ============================== To configure SASL with Kerberos, you need to create principals within your organization's KDC server for each component and for each host in each component. You also need to generate keytabs for these principals. The keytab files must be present on the Ansible host. .. note:: You need to set up your own Key Distribution Center (KDC) independently of the playbook and provide your own keytabs. The following example shows the Kerberos Configuration variables in the *all* section of the ``hosts.yaml`` file. This installs Kerberos packages and configures the ``/etc/krb5.conf`` file on each host. :: all: vars: ...omitted #### Kerberos Configuration #### kerberos_kafka_broker_primary: kerberos_configure: kerberos: realm: kdc_hostname: admin_hostname: .. note:: If the hosts already have the ``/etc/krb5.conf`` file configured, make sure to set ``kerberos_configure`` to false. The following examples show the variables that need to be added to the ``hosts.yml`` file for each component to use Kerberos for authentication. Each host must have these variables set in the ``hosts.yml`` file. **Zookeeper section** Use the following example to enter the ``zookeeper_kerberos_keytab_path`` and ``zookeeper_kerberos_principal`` variables. Each host needs to have these two variables. Three hosts are shown in the example. :: zookeeper: hosts: ip-192-24-34-224.us-west.compute.internal: zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM ip-192-24-37-15.us-west.compute.internal: zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM ip-192-24-34-224.us-west.compute.internal: zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM **Kafka broker section** Use the following example to enter the ``kafka_broker_kerberos_keytab_path`` and ``kafka_broker_kerberos_principal`` variables. Each host needs to have these two variables. The example shows three hosts. :: kafka_broker: hosts: ip-192-24-34-224.us-west.compute.internal: kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM ip-192-24-37-15.us-west.compute.internal: kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM ip-192-24-34-224.us-west.compute.internal: kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM **Schema Registry section** Use the following example to enter the ``schema_registry_kerberos_keytab_path`` and ``schema_registry_kerberos_principal`` variables. Each host needs to have these two variables. The example shows one host. :: schema_registry: hosts: ip-192-24-34-224.us-west.compute.internal: schema_registry_kerberos_keytab_path: /tmp/keytabs/schemaregistry-ip-192-24-34-224.us-west.compute.internal.keytab schema_registry_kerberos_principal: schemaregistry/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM **Kafka Connect section** Use the following example to enter the ``kafka_connect_kerberos_keytab_path`` and ``kafka_connect_kerberos_principal`` variables. Each host needs to have these two variables. The example shows one host. :: kafka_connect: hosts: ip-192-24-34-224.us-west.compute.internal: kafka_connect_kerberos_keytab_path: /tmp/keytabs/connect-ip-192-24-34-224.us-west.compute.internal.keytab kafka_connect_kerberos_principal: connect/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM **REST proxy section** Use the following example to enter the ``kafka_rest_kerberos_keytab_path`` and ``kafka_rest_kerberos_principal`` variables. Each host needs to have these two variables. The example shows one host. :: kafka_rest: hosts: ip-192-24-34-224.us-west.compute.internal: kafka_rest_kerberos_keytab_path: /tmp/keytabs/restproxy-ip-192-24-34-224.us-west.compute.internal.keytab kafka_rest_kerberos_principal: restproxy/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM **KSQL section** Use the following example to enter the ``ksql_kerberos_keytab_path`` and ``ksql_kerberos_principal`` variables. Each host needs to have these two variables. The example shows one host. :: ksql: hosts: ip-192-24-34-224.us-west.compute.internal: ksql_kerberos_keytab_path: /tmp/keytabs/ksql-ip-192-24-34-224.us-west.compute.internal.keytab ksql_kerberos_principal: ksql/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM **Control Center section** Use the following example to enter the ``control_center_kerberos_keytab_path`` and ``control_center_kerberos_principal`` variables. Each host needs to have these two variables. The example shows one host. :: control_center: hosts: ip-192-24-34-224.us-west.compute.internal: control_center_kerberos_keytab_path: /tmp/keytabs/controlcenter-ip-192-24-34-224.us-west.compute.internal.keytab control_center_kerberos_principal: controlcenter/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM ----------------------------- Migrating to Confluent Server ----------------------------- |cp| 5.3 (and later) includes a commercial component named ``confluent-server``. |cs| is a |ak| broker with support for commercial features like :ref:`Confluent Operator `, the |ak| Java client, :ref:`kafka_streams`, and :ref:`kafka_connect`. |cs| also includes proprietary plugins for the security features :ref:`kafka_ldap_authorizer` and :ref:`Role-based access control `. |cp| also includes a version of ``confluent-kafka``, which is an open source Apache 2.0 licensed component. Both ``confluent-community`` and ``confluent-platform`` packages have a dependency on ``confluent-kafka``. You can migrate from ``confluent-kafka`` to ``confluent-server`` using the instructions in the following two sections. Red Hat Enterprise Linux ^^^^^^^^^^^^^^^^^^^^^^^^ Complete the following steps to migrate to ``confluent-server`` when using Red Hat Enterprise Linux (RHEL). #. Install your cluster using cp-ansible. This installs ``confluent-kafka`` by default. #. Start your cluster and verify that it is working as expected. #. Log in and stop each |ak| broker host. :: sudo systemctl stop confluent-kafka #. As a precaution, back up the following configuration files: :: sudo cp /etc/kafka/server.properties /tmp sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp If running with Kerberos, you must back up your JAAS file: :: sudo cp /etc/kafka/kafka_server_jaas.conf /tmp #. Use ``yum swap`` to change to ``confluent-server``: :: yum swap confluent-kafka-2.12 confluent-server #. Copy your |ak| broker configuration backup to the broker configuration directory: :: sudo cp /tmp/server.properties /etc/kafka/ #. Copy your override file using the following command: :: sudo cp /tmp/override.con /etc/systemd/system/confluent-server.service.d/ .. note:: If you are running Kerberos, complete the next step. If not, skip the next step. #. Copy your JAAS file to the broker configuration directory: :: sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/ #. Start your |ak| brokers: :: sudo systemctl start confluent-kafka Complete the following steps to change from ``confluent-server`` back to ``confluent-kafka``. #. Log in to each |ak| broker host and stop each broker: :: sudo systemctl stop confluent-kafka #. As a precaution, back up the following configuration files: :: sudo cp /etc/kafka/server.properties /tmp sudo cp /etc/systemd/system/confluent-server.service.d/override.conf /tmp If running with Kerberos, you must back up your JAAS file: :: sudo cp /etc/kafka/kafka_server_jaas.conf /tmp #. Use ``yum swap`` to change to ``confluent-kafka``: :: yum swap confluent-kafka-2.12 confluent-kafka #. Copy your configuration files to the broker configuration directory: :: sudo cp /tmp/server.properties /etc/kafka/ .. note:: If you are running Kerberos, complete the next step. If not, skip the next step. #. Copy your JAAS file to the broker configuration directory: :: sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/ #. Start your |ak| brokers: :: sudo systemctl start confluent-kafka Ubuntu ^^^^^^ Complete the following steps to migrate to ``confluent-server`` when using Ubuntu. #. Install your cluster using cp-ansible. This installs ``confluent-kafka`` by default. #. Start your cluster and verify that it is working as expected. #. Log in and stop each |ak| broker host. :: sudo systemctl stop confluent-kafka #. As a precaution, back up the following configuration files: :: sudo cp /etc/kafka/server.properties /tmp sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp If running with Kerberos, you must back up your JAAS file: :: sudo cp /etc/kafka/kafka_server_jaas.conf /tmp #. Remove ``confluent-kafka`` from your |ak| broker hosts. :: sudo apt-get remove confluent-kafka-2.12 #. Install ``confluent-server`` on your |ak| broker hosts. :: sudo apt-get install confluent-server #. Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files. .. note:: If you are running Kerberos, complete the next step. If not, skip the next step. #. Copy your override file using the following command: :: sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /etc/systemd/system/confluent-server.service.d #. Start your |ak| brokers: :: sudo systemctl start confluent-kafka Complete the following steps to change from ``confluent-server`` back to ``confluent-kafka``. #. Log in and stop each |ak| broker host. :: sudo systemctl stop confluent-kafka #. As a precaution, back up the following configuration files: :: sudo cp /etc/kafka/server.properties /tmp sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp If running with Kerberos, you need to back up your JAAS file: :: sudo cp /etc/kafka/kafka_server_jaas.conf /tmp #. Remove ``confluent-server`` from your |ak| broker hosts. :: sudo apt-get remove confluent-server #. Install ``confluent-kafka`` on your |ak| broker hosts. :: sudo apt-get install confluent-kafka-2.12 #. Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files. #. Start your |ak| brokers. :: sudo systemctl start confluent-kafka ---------------------- Additional information ---------------------- This repository makes use of the `systemd scripts provided with the Confluent Platform `_. For this reason, there is an expected default user/service mapping that uses the prefix convention ``cp-`` followed by the service name. For example ``cp-kafka`` or ``cp-schema-registry``. The one exception is that |zk-full| runs as the ``cp-kafka`` user. This matches the systemd scripts as well. --------------- Troubleshooting --------------- The playbook may fail to run for a variety of reasons. Complete the following steps if the playbook fails: #. Append ``-vvv`` to the playbook run command and pipe it to a file. :: ansible-playbook -vvvv -i hosts.yml all.yml >failure.txt #. Open a support ticket with `Confluent Support `_ and provide the following: * Playbook name you are running. * The step where the playbook failed. * All changes you made to the playbook. * Attach the output from the failed test as a compressed text file.