Authentication using SASL

Prerequisites

Kerberos

If your organization is already using a Kerberos server (for example, by using Active Directory), there is no need to install a new server just for Kafka. Otherwise you will need to install one, your Linux vendor likely has packages for Kerberos and a short guide on how to install and configure it (Ubuntu, Redhat). Note that if you are using Oracle Java, you will need to download JCE policy files for your Java version and copy them to $JAVA_HOME/jre/lib/security.

Kerberos Principals

If you are using the organization’s Kerberos or Active Directory server, ask your Kerberos administrator for a principal for each Kafka broker in your cluster and for every operating system user that will access Kafka with Kerberos authentication (via clients and tools).

If you have installed your own Kerberos, you will need to create these principals yourself using the following commands:

sudo /usr/sbin/kadmin.local -q 'addprinc -randkey kafka/{hostname}@{REALM}'
sudo /usr/sbin/kadmin.local -q "ktadd -k /etc/security/keytabs/{keytabname}.keytab kafka/{hostname}@{REALM}"

All hosts must be reachable using hostnames

It is a Kerberos requirement that all your hosts can be resolved with their FQDNs.

Configuring Kafka Brokers

1. Add a suitably modified JAAS file similar to the one below to each Kafka broker’s config directory, let’s call it kafka_server_jaas.conf for this example (note that each broker should have its own keytab):

KafkaServer {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka_server.keytab"
    principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};

# Zookeeper client authentication
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka_server.keytab"
    principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};

2. Pass the name of the JAAS file as a JVM parameter to each Kafka broker:

-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf

You may also wish to specify the path to the krb5.conf file (see JDK’s Kerberos Requirements for more details):

-Djava.security.krb5.conf=/etc/kafka/krb5.conf

3. Make sure the keytabs configured in the JAAS file are readable by the operating system user who is starting the Kafka broker.
4. Configure a SASL port in server.properties, by adding at least one of SASL_PLAINTEXT or SASL_SSL to the listeners parameter, which contains one or more comma-separated values:

listeners=SASL_PLAINTEXT://host.name:port

If SASL_SSL is used, then SSL must also be configured. Note that a PLAINTEXT port is needed if you intend to use any client that does not support SASL.

If you are only configuring a SASL port (or if you want the Kafka brokers to authenticate each other using SASL) then make sure you set the same SASL protocol for inter-broker communication:

security.inter.broker.protocol=SASL_PLAINTEXT (or SASL_SSL)

We must also configure the service name in server.properties, which should match the principal name of the Kafka brokers. In the above example, the principal is kafka/kafka1.hostname.com@EXAMPLE.com, so:

sasl.kerberos.service.name="kafka"

Important notes:

1. KafkaServer is a section name in the JAAS file used by each broker. This section tells the broker which principal to use and the location of the keytab where this principal is stored. It allows the broker to login using the keytab specified in this section.
2. The Client section is used to authenticate a SASL connection with zookeeper. It also allows the brokers to set SASL ACL on ZooKeeper nodes which locks these nodes down so that only the brokers can modify it. It is necessary to have the same principal name across all brokers. If you want to use a section name other than Client, set the system property zookeeper.sasl.client to the appropriate name (e.g. -Dzookeeper.sasl.client=ZkClient).
3. ZooKeeper uses zookeeper as the service name by default. If you want to change this, set the system property zookeeper.sasl.client.username to the appropriate name (e.g. -Dzookeeper.sasl.client.username=zk).

Configuring Kafka Clients

SASL authentication is only supported for the new Kafka producer and consumer, the older API is not supported. To configure SASL authentication on the clients:

1. Clients (producers, consumers, connect workers, etc) will authenticate to the cluster with their own principal (usually with the same name as the user running the client), so obtain or create these principals as needed. Then create a JAAS file for each principal. The KafkaClient section describes how the clients like producer and consumer can connect to the Kafka Broker. The following is an example configuration for a client using a keytab (recommended for long-running processes):

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="/etc/security/keytabs/kafka_client.keytab"
    principal="kafka-client-1@EXAMPLE.COM";
};

For command-line utilities like kafka-console-consumer or kafka-console-producer, kinit can be used along with useTicketCache=true as in:

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=true;
};

2. Pass the name of the JAAS file as a JVM parameter to the client JVM:

-Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf

You may also wish to specify the path to the krb5.conf file (see JDK’s Kerberos Requirements for more details).

-Djava.security.krb5.conf=/etc/kafka/krb5.conf

3. Make sure the keytabs configured in the kafka_client_jaas.conf are readable by the operating system user who is starting kafka client.
4. Configure the following properties in producer.properties and/or consumer.properties:

security.protocol=SASL_PLAINTEXT (or SASL_SSL)
sasl.kerberos.service.name="kafka"