Install using Ansible¶
Refer to the following sections for information about using Ansible playbooks provided by Confluent to install the Confluent Platform.
Important
Support: For Confluent Platform 5.3 (and later), the Ansible playbooks are fully supported for those with a Confluent Support contract. Any issues encountered can be reported to Confluent Support at https://support.confluent.io. For those without a Confluent Support contract, issues can be reported using the cp-ansible GitHub repo and are addressed on a best-effort basis.
Feature recommendations: For those with a Confluent Support contract, future work and additional features should be filed by opening a case with Confluent Support at https://support.confluent.io. For those without a Confluent Support contract, please review the Contributing document.
About the Ansible playbooks¶
Ansible provides a simple way to deploy, manage, and configure Confluent Platform. The cp-ansible repository provides the playbooks and templates that allow you to easily provision the Confluent Platform in your environment.
Note
This guide assumes basic knowledge of how Ansible playbooks work. If you want to learn more about Ansible playbooks, see these resources:
The Ansible playbooks provided by Confluent perform the following operations:
- Installs Confluent Platform packages.
- Starts services using systemd scripts.
- Provides variables for configuring various security settings between Confluent Platform component services:
- SSL (one way and two way) with self-signed certs or custom certs
- SASL (Kerberos, Plain)
You can install the services listed below using the Ansible playbooks provided by Confluent:
- Apache ZooKeeper™
- Apache Kafka®
- Confluent Schema Registry
- REST Proxy
- Confluent Control Center
- Kafka Connect (distributed mode)
- Confluent KSQL
Note
With the exception of Confluent Control Center, all Confluent Platform components are installed with Jolokia enabled and metrics exposed.
Installation Requirements¶
Refer to the following sections for installation requirements.
General Requirements¶
- Confluent Platform 5.3 or later
- Ansible 2.7 or later (on control node)
- Confluent Platform Ansible playbooks
- SSH access between Ansible hosts and all other hosts
Supported Operating Systems¶
- RHEL 7 or later
- Ubuntu 16.04 or later
- Debian 8 or later
Hardware Requirements¶
Refer to the Confluent Platform component on-premises system requirements.
Ports¶
The playbook configures Confluent Platform components to use the ports listed below.
Important
Make sure these ports are open.
Component | Port |
---|---|
Kafka brokers: inter-broker communication | 9091 |
Kafka brokers: client communication | 9092 |
Confluent Control Center | 9021 |
Connect REST API | 8083 |
KSQL REST API | 8088 |
REST Proxy | 8082 |
Schema Registry REST API | 8081 |
ZooKeeper | 2181, 2888, 3888 |
Running the playbook¶
Using Ansible to install and configure Confluent Platform consists of a few steps. You create an inventory file, set up the variables you need, and then run the playbook. The following steps provide a high-level overview of the procedure.
Get the source code.
git clone https://github.com/confluentinc/cp-ansible
Create a copy of the
hosts_example.yml
file.cd cp-ansible cp hosts_example.yml hosts.yml
Edit your inventory file and enter the correct hosts, SSH variables, and custom variables.
vi hosts.yml
Confirm Ansible can connect over SSH.
ansible -i hosts.yml all -m ping
Run the
all.yml
playbook.ansible-playbook -i hosts.yml all.yml
The following sections provide detailed information about generating your inventory file, setting custom variables, and configuring different options for security.
Generating your inventory file¶
Before running the Ansible playbook you need to generate an inventory file. The inventory file provisions the hosts used for the Confluent Platform components. See Inventory basics: hosts and groups for more information about the inventory file.
An example base inventory file is shown below:
zookeeper:
hosts:
ip-192-24-10-207.us-west.compute.internal:
ip-192-24-5-30.us-west.compute.internal:
ip-192-24-10-0.us-west.compute.internal:
kafka_broker:
hosts:
ip-192-24-10-207.us-west.compute.internal:
broker_id: 1
ip-192-24-5-30.us-west.compute.internal:
broker_id: 2
ip-192-24-10-0.us-west.compute.internal:
broker_id: 3
schema_registry:
hosts:
ip-192-24-10-207.us-west.compute.internal:
control_center:
hosts:
ip-192-24-12-225.us-west.compute.internal:
kafka_connect:
hosts:
ip-192-24-10-207.us-west.compute.internal:
kafka_rest:
hosts:
ip-192-24-10-207.us-west.compute.internal:
ksql:
hosts:
ip-192-24-5-30.us-west.compute.internal:
Save your inventory file onto the host running Ansible as hosts.yml
. Enter
the following command to verify that Ansible can reach each host:
ansible -i /path/to/hosts.yml all -m ping
This verifies that the Ansible bastion can reach each host over SSH. This is a requirement for running the playbook. If ping fails, you likely need to set up your ssh
variables. See Connecting to hosts: behavioral inventory parameters for more information about setting up ssh
variables.
Setting up custom variables¶
Confluent provides a sample inventory file with all supported customization variables. Use this file as a reference when setting your configuration properties and when setting up your TLS an SASL configuration variables.
You can set custom properties for each component in the inventory file using variables. An example is shown below:
all:
vars:
zookeeper:
properties:
initLimit: 6
syncLimit: 3
kafka_broker:
properties:
num.io.threads: 15
In the example above, the initLimit
and syncLimit
properties are set in
the ZooKeeper properties file. The num.io.threads
property gets set in the Kafka
properties file.
JMS Exporter¶
Additionally, a variable is available that allows you to enable JMX Exporter on your Kafka brokers. The following example shows how to enable this property.
## JMX Exporter is disabled by default. When enabled, JMX Exporter jar will be pulled from the Internet and enabled on the broker *only*.
## To enable, uncomment this line:
jmxexporter_enabled: true
Configuring Security¶
The following sections provide instructions for configuring security.
TLS encryption¶
By default the components are installed using the PLAINTEXT protocol. This means there is no data encryption. You can enable TLS encryption with one of the following options:
- Self Signed Certs: A Certificate Authority will be generated by the playbooks and used to sign the certs for each host.
- Custom Certs: You can provide signed certs and keys for each host as well as the Certificate Authority Cert used to sign the certs.
- Custom Keystores and Truststores: You can provide keystores and truststores for each host.
Note
By default the TLS connection is one-way TLS. Mutual TLS is also configurable.
TLS encryption variables should be added under the all group in the hosts.yml file. To turn on TLS encryption set the following parameter to true:
ssl_enabled: true
By default, components are configured with one-way TLS. To turn on TLS mutual authentication, set the following parameter to true:
ssl_mutual_auth_enabled: true
By default, the certs for this configuration are self-signed. To deploy custom certificates, there are two options available: Custom Certificates and Custom Keystores and Truststores.
Note
The ssl_enabled
property applies globally to all Confluent Platform components
except ZooKeeper. Starting from the 5.4 release of the Confluent Platform playbooks, this
global setting will apply to ZooKeeper as well.
Custom Certificates¶
To provide Custom Certificates for each host, you will need the Certificate Authority certificate, the signed certificates, and keys for each host on the Ansible host. Complete the following steps to use Custom Certificates.
Set this variable to true.
ssl_custom_certs: true
Enter the path to the Certificate Authority Cert used to sign each host certificate.
ssl_ca_cert_filepath: "/tmp/certs/ca.crt"
Set the following variables for each host.
ssl_signed_cert_filepath: "/tmp/certs/{{inventory_hostname}}-signed.crt" ssl_key_filepath: "/tmp/certs/{{inventory_hostname}}-key.pem"
The variable
{{inventory_hostname}}
in the example shows that Ansible can read the hostnames set in the inventory file. For this reason, you can keep the inventory file shorter if you put the hostname in the filename for each signed certificate and key file.As an alternative, you can set the variables directly under a host:
schema_registry: hosts: ip-192-24-10-207.us-west.compute.internal: ssl_signed_cert_filepath: "/tmp/certs/192-24-10-207-signed.crt ssl_key_filepath: "/tmp/certs/192-24-10-207-key.pem
Custom Keystores and Truststores¶
To provide Custom Keystores and Truststores for each host, you will need to have keystores and truststores for each host on the Ansible host, as well as their passwords. Complete the following steps to use Custom Keystores and Truststores.
Set this variable to true.
provided_keystore_and_truststore: true
Set the following variables for each host.
ssl_keystore_filepath: "/tmp/certs/{{inventory_hostname}}-keystore.jks" ssl_keystore_key_password: mystorepassword ssl_keystore_store_password: mystorepassword ssl_truststore_filepath: "/tmp/certs/truststore.jks" ssl_truststore_password: truststorepass
By taking advantage of the {{inventory_hostname}}
variable and setting the
same password for each host, you can set these variable once under the all
group in the hosts.yml
file. As an alternative, you can set these variables
under each host as shown in the alternative Custom Certificates example.
SASL Authentication¶
Confluent Ansible can configure SASL/PLAIN or SASL/GSSAPI (or Kerberos). By default, there is no SASL authentication.
SASL authentication options¶
By default the components are installed without any SASL authentication. You can enable the following SASL options:
- PLAIN: SASL/PLAIN uses a simple username and password for authentication.
- SCRAM: SASL/SCRAM uses usernames and password stored in ZooKeeper. Credentials get created during installation.
- Kerberos: SASL with Kerberos provides authentication using your Kerberos or Active Directory server.
Note
Kerberos Key Distribution Center (KDC) and Active Directory KDC configurations are not currently configured by the Ansible playbook.
Configuring SASL/PLAIN¶
To configure SASL/PLAIN security, set the following option in the all group in the hosts.yml
file.
sasl_protocol: plain
During installation, users are created for each component. This includes an
admin user for the Kafka brokers and a client user for use by external
components. To configure additional users, add the following section to the
all group in the hosts.yml
file. The following example shows an additional
three users added.
sasl_plain_users:
user1:
principal: user1
password: my-secret
user2:
principal: user2
password: my-secret
user3:
principal: user3
password: my-secret
Configuring SASL/SCRAM¶
To configure SASL/SCRAM security, set the following option in the all group in the hosts.yml
file.
sasl_protocol: scram
During installation, users are created for each component. This includes an
admin user for the Kafka brokers and a client user for use by external
components. To configure additional users, add the following section to the
all group in the hosts.yml
file.
sasl_scram_users:
user1:
principal: user1
password: my-secret
Configuring SASL with Kerberos¶
To configure SASL with Kerberos, you need to create principals within your organization’s KDC server for each component and for each host in each component. You also need to generate keytabs for these principals. The keytab files must be present on the Ansible host.
Note
You need to set up your own Key Distribution Center (KDC) independently of the playbook and provide your own keytabs.
The following example shows the Kerberos Configuration variables in the all
section of the hosts.yaml
file. This installs Kerberos packages and
configures the /etc/krb5.conf
file on each host.
all:
vars:
...omitted
#### Kerberos Configuration ####
kerberos_kafka_broker_primary: <Name of the primary set on the kafka brokers principal (eg. kafka)>
kerberos_configure: <Boolean for ansible to install kerberos packages and configure this file: /etc/krb5.conf, defaults to true>
kerberos:
realm: <KDC server realm eg. confluent.example.com>
kdc_hostname: <hostname of machine with KDC running eg. ip-192-24-45-82.us-west.compute.internal>
admin_hostname: <hostname of machine with KDC running eg. ip-192-24-45-82.us-west.compute.internal>
Note
If the hosts already have the /etc/krb5.conf
file configured, make sure to set kerberos_configure
to false.
The following examples show the variables that need to be added to the
hosts.yml
file for each component to use Kerberos for authentication. Each
host must have these variables set in the hosts.yml
file.
Zookeeper section
Use the following example to enter the zookeeper_kerberos_keytab_path
and
zookeeper_kerberos_principal
variables. Each host needs to have these two
variables. Three hosts are shown in the example.
zookeeper:
hosts:
ip-192-24-34-224.us-west.compute.internal:
zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
ip-192-24-37-15.us-west.compute.internal:
zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
ip-192-24-34-224.us-west.compute.internal:
zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
Kafka broker section
Use the following example to enter the kafka_broker_kerberos_keytab_path
and
kafka_broker_kerberos_principal
variables. Each host needs to have these two
variables. The example shows three hosts.
kafka_broker:
hosts:
ip-192-24-34-224.us-west.compute.internal:
kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
ip-192-24-37-15.us-west.compute.internal:
kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
ip-192-24-34-224.us-west.compute.internal:
kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
Schema Registry section
Use the following example to enter the schema_registry_kerberos_keytab_path
and schema_registry_kerberos_principal
variables. Each host needs to have
these two variables. The example shows one host.
schema_registry:
hosts:
ip-192-24-34-224.us-west.compute.internal:
schema_registry_kerberos_keytab_path: /tmp/keytabs/schemaregistry-ip-192-24-34-224.us-west.compute.internal.keytab
schema_registry_kerberos_principal: schemaregistry/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
Kafka Connect section
Use the following example to enter the kafka_connect_kerberos_keytab_path
and kafka_connect_kerberos_principal
variables. Each host needs to have
these two variables. The example shows one host.
kafka_connect:
hosts:
ip-192-24-34-224.us-west.compute.internal:
kafka_connect_kerberos_keytab_path: /tmp/keytabs/connect-ip-192-24-34-224.us-west.compute.internal.keytab
kafka_connect_kerberos_principal: connect/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
REST proxy section
Use the following example to enter the kafka_rest_kerberos_keytab_path
and
kafka_rest_kerberos_principal
variables. Each host needs to have these two
variables. The example shows one host.
kafka_rest:
hosts:
ip-192-24-34-224.us-west.compute.internal:
kafka_rest_kerberos_keytab_path: /tmp/keytabs/restproxy-ip-192-24-34-224.us-west.compute.internal.keytab
kafka_rest_kerberos_principal: restproxy/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
KSQL section
Use the following example to enter the ksql_kerberos_keytab_path
and
ksql_kerberos_principal
variables. Each host needs to have these two
variables. The example shows one host.
ksql:
hosts:
ip-192-24-34-224.us-west.compute.internal:
ksql_kerberos_keytab_path: /tmp/keytabs/ksql-ip-192-24-34-224.us-west.compute.internal.keytab
ksql_kerberos_principal: ksql/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
Control Center section
Use the following example to enter the control_center_kerberos_keytab_path
and control_center_kerberos_principal
variables. Each host needs to have
these two variables. The example shows one host.
control_center:
hosts:
ip-192-24-34-224.us-west.compute.internal:
control_center_kerberos_keytab_path: /tmp/keytabs/controlcenter-ip-192-24-34-224.us-west.compute.internal.keytab
control_center_kerberos_principal: controlcenter/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
Security for Internal and External Listeners¶
Confluent Ansible configures two listeners on the broker: an internal listener
for the broker to communicate with other brokers and an external listener for
the other Confluent Platform components and external clients. By default both of these
listeners inherit the security settings you configure for ssl_enabled
and
sasl_protocol
.
If you only need a single listener, add the following variable to the all
group in hosts.yml
file.
kafka_broker_configure_additional_brokers: false
To configure different security settings for the internal and external listeners, add the following variables to the all group in the hosts.yml
file.
kafka_broker_custom_listeners:
internal:
name: INTERNAL
port: 9091
ssl_enabled: false
ssl_mutual_auth_enabled: false
sasl_protocol: none
external:
name: EXTERNAL
port: 9092
ssl_enabled: true
ssl_mutual_auth_enabled: false
sasl_protocol: scram
You can create additional custom listeners within the kafka_broker_custom_listeners
section shown in the example above. Simply add another listener block. The
example below shows an additional custom client listener.
client_listener:
name: CLIENT
port: 9093
ssl_enabled: true
ssl_mutual_auth_enabled: true
sasl_protocol: plain
Migrating to Confluent Server¶
Confluent Platform 5.3 (and later) includes a commercial component named confluent-server
.
Confluent Server is a Kafka broker with support for commercial features like Confluent
Operator, the Kafka Java client, Kafka Streams,
and Kafka Connect. Confluent Server also includes proprietary plugins for the
security features Confluent LDAP Authorizer and Role-based access
control.
Confluent Platform also includes a version of confluent-kafka
, which is an open source
Apache 2.0 licensed component. Both confluent-community
and
confluent-platform
packages have a dependency on confluent-kafka
. You
can migrate from confluent-kafka
to confluent-server
using the
instructions in the following two sections.
Red Hat Enterprise Linux¶
Complete the following steps to migrate to confluent-server
when using Red
Hat Enterprise Linux (RHEL).
Install your cluster using cp-ansible. This installs
confluent-kafka
by default.Start your cluster and verify that it is working as expected.
Log in and stop each Kafka broker host.
sudo systemctl stop confluent-kafka
As a precaution, back up the following configuration files:
sudo cp /etc/kafka/server.properties /tmp sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp
If running with Kerberos, you must back up your JAAS file:
sudo cp /etc/kafka/kafka_server_jaas.conf /tmp
Use
yum swap
to change toconfluent-server
:yum swap confluent-kafka-2.12 confluent-server
Copy your Kafka broker configuration backup to the broker configuration directory:
sudo cp /tmp/server.properties /etc/kafka/
Copy your override file using the following command:
sudo cp /tmp/override.con /etc/systemd/system/confluent-server.service.d/
Note
If you are running Kerberos, complete the next step. If not, skip the next step.
Copy your JAAS file to the broker configuration directory:
sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/
Start your Kafka brokers:
sudo systemctl start confluent-server
Complete the following steps to change from confluent-server
back to
confluent-kafka
.
Log in to each Kafka broker host and stop each broker:
sudo systemctl stop confluent-server
As a precaution, back up the following configuration files:
sudo cp /etc/kafka/server.properties /tmp sudo cp /etc/systemd/system/confluent-server.service.d/override.conf /tmp
If running with Kerberos, you must back up your JAAS file:
sudo cp /etc/kafka/kafka_server_jaas.conf /tmp
Use
yum swap
to change toconfluent-kafka
:yum swap confluent-kafka-2.12 confluent-kafka
Copy your configuration files to the broker configuration directory:
sudo cp /tmp/server.properties /etc/kafka/
Note
If you are running Kerberos, complete the next step. If not, skip the next step.
Copy your JAAS file to the broker configuration directory:
sudo cp /tmp/kafka_server_jaas.conf /etc/kafka/
Start your Kafka brokers:
sudo systemctl start confluent-kafka
Ubuntu¶
Complete the following steps to migrate to confluent-server
when using Ubuntu.
Install your cluster using cp-ansible. This installs
confluent-kafka
by default.Start your cluster and verify that it is working as expected.
Log in and stop each Kafka broker host.
sudo systemctl stop confluent-kafka
As a precaution, back up the following configuration files:
sudo cp /etc/kafka/server.properties /tmp sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp
If running with Kerberos, you must back up your JAAS file:
sudo cp /etc/kafka/kafka_server_jaas.conf /tmp
Remove
confluent-kafka
from your Kafka broker hosts.sudo apt-get remove confluent-kafka-2.12
Install
confluent-server
on your Kafka broker hosts.sudo apt-get install confluent-server
Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files.
Note
If you are running Kerberos, complete the next step. If not, skip the next step.
Copy your override file using the following command:
sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /etc/systemd/system/confluent-server.service.d
Start your Kafka brokers:
sudo systemctl start confluent-server
Complete the following steps to change from confluent-server
back to
confluent-kafka
.
Log in and stop each Kafka broker host.
sudo systemctl stop confluent-server
As a precaution, back up the following configuration files:
sudo cp /etc/kafka/server.properties /tmp sudo cp /etc/systemd/system/confluent-kafka.service.d/override.conf /tmp
If running with Kerberos, you need to back up your JAAS file:
sudo cp /etc/kafka/kafka_server_jaas.conf /tmp
Remove
confluent-server
from your Kafka broker hosts.sudo apt-get remove confluent-server
Install
confluent-kafka
on your Kafka broker hosts.sudo apt-get install confluent-kafka-2.12
Select the default option to keep your configuration files in place. Get the backup configuration files (that you made earlier) if you accidentally replace the files.
Start your Kafka brokers.
sudo systemctl start confluent-kafka
Additional information¶
This repository uses the systemd scripts provided with the Confluent Platform. Because of this dependency, there is an expected default user/service mapping that uses the prefix convention cp-
followed by the service name. For example cp-kafka
or cp-schema-registry
. The one exception is that ZooKeeper runs as the cp-kafka
user. This matches the systemd scripts as well.
Troubleshooting¶
The playbook may fail to run for a variety of reasons. Complete the following steps if the playbook fails:
Append
-vvv
to the playbook run command and pipe it to a file.ansible-playbook -vvvv -i hosts.yml all.yml >failure.txt
Open a support ticket with Confluent Support and provide the following:
- Playbook name you are running.
- The step where the playbook failed.
- All changes you made to the playbook.
- Attach the output from the failed test as a compressed text file.