Important Warning

Update July 2016: This Tech Preview documentation from March 2016 is outdated and deprecated. Please use the latest Confluent Platform documentation instead.

Authentication using SASL

Prerequisites

Kerberos

If your organization is already using a Kerberos server (for example, by using Active Directory), there is no need to install a new server just for Kafka. Otherwise you will need to install one, your Linux vendor likely has packages for Kerberos and a short guide on how to install and configure it (Ubuntu, Redhat). Note that if you are using Oracle Java, you will need to download JCE policy files for your Java version and copy them to $JAVA_HOME/jre/lib/security.

Kerberos Principals

If you are using the organization’s Kerberos or Active Directory server, ask your Kerberos administrator for a principal for each Kafka broker in your cluster and for every operating system user that will access Kafka with Kerberos authentication (via clients and tools).

If you have installed your own Kerberos, you will need to create these principals yourself using the following commands:

sudo /usr/sbin/kadmin.local -q 'addprinc -randkey kafka/{hostname}@{REALM}'
sudo /usr/sbin/kadmin.local -q "ktadd -k /etc/security/keytabs/{keytabname}.keytab kafka/{hostname}@{REALM}"

All hosts must be reachable using hostnames

It is a Kerberos requirement that all your hosts can be resolved with their FQDNs.

Configuring Kafka Brokers

  1. Add a suitably modified JAAS file similar to the one below to each Kafka broker’s config directory, let’s call it kafka_server_jaas.conf for this example (note that each broker should have its own keytab):
KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka_server.keytab"
    principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};

// ZooKeeper client authentication
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka_server.keytab"
    principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};
  1. Pass the name of the JAAS file as a JVM parameter to each Kafka broker:
-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf

You may also wish to specify the path to the krb5.conf file (see JDK’s Kerberos Requirements for more details):

-Djava.security.krb5.conf=/etc/kafka/krb5.conf
  1. Make sure the keytabs configured in the JAAS file are readable by the operating system user who is starting the Kafka broker.
  2. Configure a SASL port in server.properties, by adding at least one of SASL_PLAINTEXT or SASL_SSL to the listeners and optionally advertised.listeners properties, each of which should contain one or more comma-separated values:
listeners=SASL_PLAINTEXT://host.name:port
# The following is only needed if the value is different from ``listeners``, but it should contain
# the same security protocols as ``listeners``
advertised.listeners=SASL_PLAINTEXT://host.name:port

If SASL_SSL is used, then SSL must also be configured. Note that a PLAINTEXT port is needed if you intend to use any client that does not support SASL.

Note that advertised.host.name and advertised.port configure a single PLAINTEXT port and are incompatible with secure protocols. Please use advertised.listeners instead.

If you are only configuring a SASL port (or if you want the Kafka brokers to authenticate each other using SASL) then make sure you set the same SASL protocol for inter-broker communication:

security.inter.broker.protocol=SASL_PLAINTEXT (or SASL_SSL)

We must also configure the service name in server.properties, which should match the primary name of the Kafka brokers. In the above example, the principal is kafka/kafka1.hostname.com@EXAMPLE.com and the primary name is kafka:

sasl.kerberos.service.name=kafka

Important notes:

  1. KafkaServer is a section name in the JAAS file used by each broker. This section tells the broker which principal to use and the location of the keytab where this principal is stored. It allows the broker to login using the keytab specified in this section.
  2. The Client section is used to authenticate a SASL connection with ZooKeeper. It also allows the brokers to set SASL ACL on ZooKeeper nodes which locks these nodes down so that only the brokers can modify it. It is necessary to have the same primary name across all brokers. If you want to use a section name other than Client, set the system property zookeeper.sasl.client to the appropriate name (e.g. -Dzookeeper.sasl.client=ZkClient).
  3. ZooKeeper uses zookeeper as the service name by default. If you want to change this, set the system property zookeeper.sasl.client.username to the appropriate name (e.g. -Dzookeeper.sasl.client.username=zk).

Configuring Kafka Clients

SASL authentication is only supported for the new Kafka producer and consumer, the older API is not supported. To configure SASL authentication on the clients:

1. Clients (producers, consumers, connect workers, etc) will authenticate to the cluster with their own principal (usually with the same name as the user running the client), so obtain or create these principals as needed. Then create a JAAS file for each principal. The KafkaClient section describes how the clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration for a client using a keytab (recommended for long-running processes):

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka_client.keytab"
    principal="kafka-client-1@EXAMPLE.COM";
};

For command-line utilities like kafka-console-consumer or kafka-console-producer, kinit can be used along with useTicketCache=true as in:

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=true;
};
  1. Pass the name of the JAAS file as a JVM parameter to the client JVM:
-Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf

You may also wish to specify the path to the krb5.conf file (see JDK’s Kerberos Requirements for more details).

-Djava.security.krb5.conf=/etc/kafka/krb5.conf
  1. Make sure the keytabs configured in the kafka_client_jaas.conf are readable by the operating system user who is starting kafka client.
  2. Configure the following properties in producer.properties and/or consumer.properties:
security.protocol=SASL_PLAINTEXT (or SASL_SSL)
sasl.kerberos.service.name=kafka

Enabling Logging for SASL

To enable SASL debug output, you can set the sun.security.krb5.debug system property to true.