Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Configuring Security

The topics in this section describes how to configure security for Ansible Playbooks for Confluent Platform.

TLS encryption

By default, the Confluent Platform components are installed using the PLAINTEXT protocol with no data encryption. You can enable TLS encryption with one of the following options:

  • Self Signed Certs: A Certificate Authority will be generated by the playbooks and used to sign the certs for each host.
  • Custom Certs: You provide signed certs and keys for each host as well as the Certificate Authority Cert used to sign the certs.
  • Custom Keystores and Truststores: You provide keystores and truststores for each host.

Note

By default the TLS connection is one-way TLS. Mutual TLS is also configurable.

Configuring TLS for all components

To enable TLS encryption for all components, add the following in the hosts.yml file.

ssl_enabled: true

TLS encryption variables should be added under the all group in the hosts.yml file.

Configuring TLS for individual components

To selectively enable or disable TLS encryption for specific components, set the following settings to true or false in addition to the global ssl_enabled setting.

  • kafka_connect_ssl_enabled
  • kafka_rest_ssl_enabled
  • schema_registry_ssl_enabled
  • control_center_ssl_enabled
  • ksql_ssl_enabled

For example, if you want TLS enabled for all components except for Schema Registry, set:

ssl_enabled: true
schema_registry_ssl_enabled: false

By default, components are configured with one-way TLS. To turn on TLS mutual authentication, set the following parameter to true:

ssl_mutual_auth_enabled: true

By default, the certs for this configuration are self-signed. To deploy custom certificates, there are two options available: Custom Certificates and Custom Keystores and Truststores.

Custom certificates

To provide custom certificates for each host, you need the Certificate Authority certificate, the signed certificates, and keys for each host on the Ansible host. Complete the following steps to use custom certificates.

  1. Set ssl_custom_certs to true.

    ssl_custom_certs: true

  2. Enter the path to the Certificate Authority Cert used to sign each host certificate.

    ssl_ca_cert_filepath: "/tmp/certs/ca.crt"

  3. Set the following variables for each host.

    ssl_signed_cert_filepath: "/tmp/certs/{{inventory_hostname}}-signed.crt"

ssl_key_filepath: "/tmp/certs/{{inventory_hostname}}-key.pem"

    The variable {{inventory_hostname}} in the example shows that Ansible can read the hostnames set in the inventory file. For this reason, you can keep the inventory file shorter if you put the hostname in the filename for each signed certificate and key file.

    As an alternative, you can set the variables directly under a host:

    schema_registry:
   hosts:
      ip-192-24-10-207.us-west.compute.internal:
         ssl_signed_cert_filepath: "/tmp/certs/192-24-10-207-signed.crt
         ssl_key_filepath: "/tmp/certs/192-24-10-207-key.pem

Custom Keystores and Truststores

To provide Custom Keystores and Truststores for each host, you will need to have keystores and truststores for each host on the Ansible host, as well as their passwords. Complete the following steps to use Custom Keystores and Truststores.

  1. Set this variable to true.

    provided_keystore_and_truststore: true

  2. Set the following variables for each host.

    ssl_keystore_filepath: "/tmp/certs/{{inventory_hostname}}-keystore.jks"
ssl_keystore_key_password: mystorepassword
ssl_keystore_store_password: mystorepassword
ssl_truststore_filepath: "/tmp/certs/truststore.jks"
ssl_truststore_password: truststorepass

Using the {{inventory_hostname}} variable and setting the same password for each host, you can set these variable once under the all group in the hosts.yml file. As an alternative, you can set these variables under each host as shown in the alternative Custom Certificates example.

SASL authentication

Confluent Ansible can configure SASL/PLAIN or SASL/GSSAPI (or Kerberos). By default, the Confluent Platform components are installed without any SASL authentication.

SASL authentication options

You can enable the following SASL options:

  • PLAIN: SASL/PLAIN uses a simple username and password for authentication.
  • SCRAM: SASL/SCRAM uses usernames and password stored in ZooKeeper. Credentials get created during installation.
  • Kerberos: SASL with Kerberos provides authentication using your Kerberos or Active Directory server.

Note

Kerberos Key Distribution Center (KDC) and Active Directory KDC configurations are not currently configured by the Ansible playbook.

Configuring SASL/PLAIN

To configure SASL/PLAIN security, set the following option in the all group in the hosts.yml file.

sasl_protocol: plain

During installation, users are created for each component. This includes an admin user for the Kafka brokers and a client user for use by external components. To configure additional users, add the following section to the all group in the hosts.yml file. The following example shows an additional three users added.

sasl_plain_users:
  user1:
    principal: user1
    password: my-secret
  user2:
    principal: user2
    password: my-secret
  user3:
    principal: user3
    password: my-secret

Configuring SASL/SCRAM

To configure SASL/SCRAM security, set the following option in the all group in the hosts.yml file.

sasl_protocol: scram

During installation, users are created for each component. This includes an admin user for the Kafka brokers and a client user for use by external components. To configure additional users, add the following section to the all group in the hosts.yml file.

sasl_scram_users:
  user1:
    principal: user1
    password: my-secret

Configuring SASL with Kerberos

To configure SASL with Kerberos, you need to create principals within your organization’s KDC server for each component and for each host in each component. You also need to generate keytabs for these principals. The keytab files must be present on the Ansible host.

Note

You need to set up your own Key Distribution Center (KDC) independently of the playbook and provide your own keytabs.

Add the following configuration parameters in in the all section of the hosts.yaml file. This installs Kerberos packages and configures the /etc/krb5.conf file on each host.

  • kerberos_kafka_broker_primary: The primary part of the Kafka broker principal. For example, for the principal, kafka/kafka1.hostname.com@EXAMPLE.COM, the primary is kafka.
  • kerberos_configure: Boolean for Ansible to install Kerberos packages and to configure the /etc/krb5.conf file. The default is true.
  • realm: The realm part of the Kafka broker Kerberos principal.
  • kdc_hostname: Hostname of machine with KDC running.
  • admin_hostname: Hostname of machine with KDC running.

The following example shows the Kerberos configuration settings for the Kerberos principal, kafka/kafka1.hostname.com@EXAMPLE.COM.

all:
  vars:

  ...omitted

  #### Kerberos Configuration ####

  kerberos_kafka_broker_primary: kafka
  kerberos_configure: true
  kerberos:
    realm: example.com
    kdc_hostname: ip-192-24-45-82.us-west.compute.internal
    admin_hostname: ip-192-24-45-82.us-west.compute.internal

Note

If the hosts already have the /etc/krb5.conf file configured, make sure to set kerberos_configure to false.

The following examples show the variables that need to be added to the hosts.yml file for each component to use Kerberos for authentication. Each host must have these variables set in the hosts.yml file.

Zookeeper section

Use the following example to enter the zookeeper_kerberos_keytab_path and zookeeper_kerberos_principal variables. Each host needs to have these two variables. Three hosts are shown in the example.

zookeeper:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-37-15.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-34-224.us-west.compute.internal:
     zookeeper_kerberos_keytab_path: /tmp/keytabs/zookeeper-ip-192-24-34-224.us-west.compute.internal.keytab
     zookeeper_kerberos_principal: zookeeper/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Kafka broker section

Use the following example to enter the kafka_broker_kerberos_keytab_path and kafka_broker_kerberos_principal variables. Each host needs to have these two variables. The example shows three hosts.

kafka_broker:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-37-15.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM
   ip-192-24-34-224.us-west.compute.internal:
     kafka_broker_kerberos_keytab_path: /tmp/keytabs/kafka-ip-192-24-34-224.us-west.compute.internal.keytab
     kafka_broker_kerberos_principal: kafka/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Schema Registry section

Use the following example to enter the schema_registry_kerberos_keytab_path and schema_registry_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

schema_registry:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      schema_registry_kerberos_keytab_path: /tmp/keytabs/schemaregistry-ip-192-24-34-224.us-west.compute.internal.keytab
      schema_registry_kerberos_principal: schemaregistry/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Kafka Connect section

Use the following example to enter the kafka_connect_kerberos_keytab_path and kafka_connect_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

kafka_connect:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      kafka_connect_kerberos_keytab_path: /tmp/keytabs/connect-ip-192-24-34-224.us-west.compute.internal.keytab
      kafka_connect_kerberos_principal: connect/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

REST proxy section

Use the following example to enter the kafka_rest_kerberos_keytab_path and kafka_rest_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

kafka_rest:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      kafka_rest_kerberos_keytab_path: /tmp/keytabs/restproxy-ip-192-24-34-224.us-west.compute.internal.keytab
      kafka_rest_kerberos_principal: restproxy/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

KSQL section

Use the following example to enter the ksql_kerberos_keytab_path and ksql_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

ksql:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      ksql_kerberos_keytab_path: /tmp/keytabs/ksql-ip-192-24-34-224.us-west.compute.internal.keytab
      ksql_kerberos_principal: ksql/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Control Center section

Use the following example to enter the control_center_kerberos_keytab_path and control_center_kerberos_principal variables. Each host needs to have these two variables. The example shows one host.

control_center:
  hosts:
    ip-192-24-34-224.us-west.compute.internal:
      control_center_kerberos_keytab_path: /tmp/keytabs/controlcenter-ip-192-24-34-224.us-west.compute.internal.keytab
      control_center_kerberos_principal: controlcenter/ip-192-24-34-224.us-west.compute.internal@REALM.EXAMPLE.COM

Security for internal and external listeners

Ansible Playbooks for Confluent Platform configures two listeners on the broker: an internal listener for the broker to communicate with other brokers and an external listener for the other Confluent Platform components and external clients. By default both of these listeners inherit the security settings you configure for ssl_enabled and sasl_protocol.

If you only need a single listener, add the following variable to the all group in hosts.yml file.

kafka_broker_configure_additional_brokers: false

To configure different security settings for the internal and external listeners, add the following variables to the all group in the hosts.yml file.

kafka_broker_custom_listeners:
  internal:
    name: INTERNAL
    port: 9091
    ssl_enabled: false
    ssl_mutual_auth_enabled: false
    sasl_protocol: none
  external:
    name: EXTERNAL
    port: 9092
    ssl_enabled: true
    ssl_mutual_auth_enabled: false
    sasl_protocol: scram

You can create additional custom listeners within the kafka_broker_custom_listeners section shown in the example above. Simply add another listener block. The example below shows an additional custom client listener.

client_listener:
  name: CLIENT
  port: 9093
  ssl_enabled: true
  ssl_mutual_auth_enabled: true
  sasl_protocol: plain