Configuring Kafka Client Authentication with LDAP

Important

This software is available under a Confluent enterprise license. You can use this software for a 30-day trial period without a license key. If you are a subscriber, contact Confluent Support at support@confluent.io for more information. Note that this license applies to Kafka brokers. Kafka clients do not require a license.

You can use Active Directory (AD) and/or LDAP to configure Kafka client authentication across all of your Kafka clusters that use SASL/PLAIN. The SASL/PLAIN binding to LDAP requires a password provided by the Kafka client. Note that you cannot bind SASL/SCRAM to LDAP because Kafka client credentials (the password) cannot be sent by the Kafka client.

You must set up an LDAP server (for example, AD) before starting up the Kafka cluster. The configuration that follows is based on the assumption that you have an LDAP server at the URL LDAPSERVER.EXAMPLE.COM:3268 that is accessible using DNS lookup from the host where the broker is run. While the configuration expects a Kerberos-enabled LDAP server, Kerberos is not required; you can perform a simple bind if your LDAP supports it. These security settings must match your LDAP server configuration.

If your LDAP server authenticates Kafka clients using Kerberos, the keytab file and principal should be updated in authorizer JAAS configuration option ldap.sasl.jaas.config.

Tip

The approach to configuring Kafka client authentication with LDAP depends in large part on the LDAP mechanism you want to use:

Kafka requests that the LDAP server validate credentials (recommended)

Using this method, Kafka takes user-specified LDAP login credentials, converts the username into an LDAP-formatted name (such as DN=JDoe,OU=eng,DC=org,DC=com), then uses the LDAP-formatted name and user-supplied password to log in to LDAP as the identified user. This is known as a simple bind. This method may result in many new connections/binds to LDAP, but is considered more secure because Kafka does not need access to user passwords stored in LDAP.

If you want to avoid local comparisons and have the LDAP server perform validation by having Kafka perform simple binds on behalf of users, then do not specify a value for ldap.user.password.attribute.

Kafka performs a local password comparison

Using this method, Kafka fetches local LDAP entries that have a hashed password in the user entry of the ldap.user.password.attribute property. Kafka takes the password supplied by the user and performs a local hash and compares it to what is stored in the LDAP user entry.

If you want Kafka to fetch user passwords stored in ldap.user.object.class from the LDAP server and perform credential validation locally, specify ldap.user.password.attribute to indicate the LDAP attribute in your schema that contains the user’s password. Note that this is not recommended, secure, or a best practice because Kafka must have permission to read hashed passwords from LDAP, and may have compatibility issues if, for example, the LDAP server uses a hash function that Kafka does not recognize.

To configure Kafka client authentication with AD/LDAP:

1. Start the LDAP server.

2. Add the user name and password to LDAP:

   dn: uid=client,ou=people,dc=planetexpress,dc=com
   userPassword: client-secret
   

3. Enable LDAP authentication for Kafka clients by adding the LDAP callback handler to server.properties in the broker.

   Add the SASL configuration:

   listener.name.sasl_plaintext.sasl.enabled.mechanisms=PLAIN
   listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required;
   
   listener.name.sasl_plaintext.plain.sasl.server.callback.handler.class=
   io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandler
   

   If you want to use LDAP authentication for interbroker communication, then you must include the broker’s user name and password in your SASL configuration.

   Add the LDAP configuration:

   ldap.java.naming.provider.url=ldap://openldap:389
   # Authenticate to LDAP
   ldap.java.naming.security.principal=CN=admin,DC=planetexpress,DC=com
   ldap.java.naming.security.credentials=GoodNewsEveryone
   ldap.java.naming.security.authentication=simple
   # Locate users
   ldap.user.search.base=ou=people,dc=planetexpress,dc=com
   ldap.user.name.attribute=uid
   ldap.user.object.class=user
   

   Note

   Only include the attribute ldap.user.password.attribute when your LDAP server does not support simple bind. Otherwise, be aware that if you include ldap.user.password.attribute, simple bind is disabled.

4. Restart the Kafka broker.

   /bin/kafka-server-start etc/kafka/server.properties
   

5. Specify the Kafka client configuration in producer.properties and consumer.properties:

   sasl.mechanism=PLAIN
   security.protocol=SASL_PLAINTEXT
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
     required username="client" password="client-secret";
   

   It’s recommended that you encrypt the password in your Kafka client configuration using Secrets Management. The following example shows an encrypted Kafka client configuration:

   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
     required username="client" password=${securepass:/secretsDemo/server.properties:server.properties/sasl.jaas.config/=org.apache.kafka.common.security.plain.PlainLoginModule /password};
   

   Note

   Credentials are sent in PLAIN text, so be sure to use TLS with LDAP.

Testing and Troubleshooting LDAP Client Authentication

This section provides basic troubleshooting tips to address common errors that can occur when configuring LDAP client authentication.

Test your LDAP client configuration using the following steps.

1. Verify LDAP connectivity:

   # Ping the LDAP host to verify connectivity
   ping ldap.hostname.com
   # Connect to the LDAP host (this command uses the default port)
   telnet ldap.hostname.com 389
   

2. Install the ldapsearch tool to conduct subsequent tests:

   sudo yum install openldap-clients -y
   

3. Verify basic access without specifying any credentials:

   ldapsearch -LLL -x -H ldap://ldap.hostname.com -s "base" -b "" supportedSASLMechanisms
   

   This command should return either a success response or an authentication error. Any other errors indicate that LDAP not set up correctly.

   A success response looks like the following:

   ldapsearch -LLL -x -H ldap://localhost -s "base" -b "" supportedSASLMechanisms
   dn:
   supportedSASLMechanisms: GS2-IAKERB
   supportedSASLMechanisms: GS2-KRB5
   supportedSASLMechanisms: SCRAM-SHA-1
   supportedSASLMechanisms: SCRAM-SHA-256
   supportedSASLMechanisms: GSS-SPNEGO
   supportedSASLMechanisms: GSSAPI
   supportedSASLMechanisms: DIGEST-MD5
   supportedSASLMechanisms: OTP
   supportedSASLMechanisms: NTLM
   supportedSASLMechanisms: CRAM-MD5
   

   An authentication error looks like the following:

   ldapsearch -LLL -x -W -H ldap://localhost -s "base" -b "" supportedSASLMechanisms
   Enter LDAP Password:
   ldap_bind: Invalid credentials (49)
   

   In cases where you are unable to communicate with an LDAP server, and/or the LDAP server is not set up correctly to accept LDAP requests, the error looks like the following:

   ldapsearch -LLL -x -H ldap://localhost:8090 -s "base" -b "" supportedSASLMechanisms
   ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
   

4. Verify basic credential-based access using the same user specified in ldap.java.naming.security.principal in server.properties:

   ldapsearch -LLL -x -H ldap://ldap.hostname.com -s "base" -b "" -D CN=kafka_user,CN=Users,DC=hostname,DC=com -w 'pa55word' supportedSASLMechanisms
   

   If the value of ldap.java.naming.security.principal points to a Kerberos principal, be sure to specify the corresponding user from LDAP.

If the preceding verification tests work, then MDS can authenticate against LDAP.

When configuring Kafka to communicate over LDAPS, verify TLS connectivity to the LDAP server as follows:

# To verify the LDAP configuration
ldapsearch -LLL -x -H ldaps://ldap.hostname.com -s "base" -b "" supportedSASLMechanisms
# If using self-signed certificates from the LDAP server
LDAPTLS_CACERT=/path/to/CA.cert ldapsearch -LLL -x -H ldaps://ldap.hostname.com -s "base" -b "" supportedSASLMechanisms

When configuring Kafka to authenticate to LDAP using Kerberos, verify authentication to the LDAP server as follows:

# To verify the Kerberos configuration
kinit -k -t ./filename.keytab kafka_broker/kafka1.hostname.com@EXAMPLE.COM
ldapsearch -LLL -Y GSSAPI -H ldap://ldap.hostname.com -s "base" -b "" supportedSASLMechanisms