Configuring Kafka Client Authentication with LDAP

Important

This software is available under a Confluent enterprise license. You can use this software for a 30-day trial period without a license key. If you are a subscriber, please contact Confluent Support at support@confluent.io for more information. Note that this license applies to Kafka brokers. Kafka clients do not require a license.

You can use Active Directory (AD) and/or LDAP to configure Kafka client authentication across all of your Kafka clusters that use SASL/PLAIN. The SASL/PLAIN binding to LDAP requires a password provided by the Kafka client. Note that you cannot bind SASL/SCRAM to LDAP because Kafka client credentials (the password) cannot be sent by the Kafka client.

You must set up an LDAP server (for example, AD) before starting up the Kafka cluster. The configuration that follows is based on the assumption that you have an LDAP server at the URL LDAPSERVER.EXAMPLE.COM:3268 that is accessible using DNS lookup from the host where the broker is run. While the configuration expects a Kerberos-enabled LDAP server, Kerberos is not required; you can perform a simple bind if your LDAP supports it. These security settings must match your LDAP server configuration.

If your LDAP server authenticates Kafka clients using Kerberos, a keytab file is required for the LDAP authorizer and the keytab file and principal should be updated in authorizer JAAS configuration option ldap.sasl.jaas.config.

Tip

The approach to configuring Kafka client authentication with LDAP depends in large part on the LDAP mechanism you want to use:

Kafka requests that the LDAP server validate credentials (recommended)

Using this method, Kafka takes user-specified LDAP login credentials, converts the username into an LDAP-formatted name (such as DN=JDoe,OU=eng,DC=org,DC=com), then uses the LDAP-formatted name and user-supplied password to log in to LDAP as the identified user. This is known as a simple bind. This method may result in many new connections/binds to LDAP, but is considered more secure because Kafka does not need access to user passwords stored in LDAP.

If you want to avoid local comparisons and have the LDAP server perform validation by having Kafka perform simple binds on behalf of users, then do not specify a value for ldap.user.password.attribute.

Kafka performs a local password comparison

Using this method, Kafka fetches local LDAP entries that have a hashed password in the user entry of the ldap.user.password.attribute property. Kafka takes the password supplied by the user and performs a local hash and compares it to what is stored in the LDAP user entry.

If you want Kafka to fetch user passwords stored in ldap.user.object.class from the LDAP server and perform credential validation locally, specify ldap.user.password.attribute to indicate the LDAP attribute in your schema that contains the user’s password. Note that this is not recommended, secure, or a best practice because Kafka must have permission to read hashed passwords from LDAP, and may have compatibility issues if, for example, the LDAP server uses a hash function that Kafka does not recognize.

To configure Kafka client authentication with AD/LDAP:

  1. Start the LDAP server.

  2. Add the user name and password to LDAP:

    dn: uid=client,ou=people,dc=planetexpress,dc=com
    userPassword: client-secret
    
  3. Enable LDAP authentication for Kafka clients by adding the LDAP callback handler to server.properties in the broker.

    Add the SASL configuration:

    listener.name.sasl_plaintext.sasl.enabled.mechanisms=PLAIN
    listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required;
    
    listener.name.sasl_plaintext.plain.sasl.server.callback.handler.class=
    io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandler
    

    If you want to use LDAP authentication for inter-broker communication, then you must include the broker’s user name and password in your SASL configuration.

    Add the LDAP configuration:

    ldap.java.naming.provider.url=ldap://openldap:389
    # Authenticate to LDAP
    ldap.java.naming.security.principal=CN=admin,DC=planetexpress,DC=com
    ldap.java.naming.security.credentials=GoodNewsEveryone
    ldap.java.naming.security.authentication=simple
    # Locate users
    ldap.user.search.base=ou=people,dc=planetexpress,dc=com
    ldap.user.name.attribute=uid
    ldap.user.object.class=user
    

    Note

    Only include the attribute ldap.user.password.attribute when your LDAP server does not support simple bind. Otherwise, be aware that if you include ldap.user.password.attribute, simple bind is disabled.

  4. Restart the Kafka broker.

    /bin/kafka-server-start etc/kafka/server.properties
    
  5. Specify the Kafka client configuration in producer.properties and consumer.properties:

    sasl.mechanism=PLAIN
    security.protocol=SASL_PLAINTEXT
    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
      required username="client" password="client-secret";
    

    It’s recommended that you encrypt the password in your Kafka client configuration using Secrets Management. The following example shows an encrypted Kafka client configuration:

    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
      required username="client" password=${securepass:/secretsDemo/server.properties:server.properties/sasl.jaas.config/=org.apache.kafka.common.security.plain.PlainLoginModule /password};
    

    Note

    Credentials are sent in PLAIN text, so be sure to use TLS with LDAP.

Testing and Troubleshooting LDAP Client Authentication

This section provides basic troubleshooting tips to address common errors that can occur when configuring LDAP client authentication.

Test your LDAP client configuration using the following steps.

  1. Verify LDAP connectivity:

    # Ping the LDAP host to verify connectivity
    ping ldap.hostname.com
    # Connect to the LDAP host (this command uses the default port)
    telnet ldap.hostname.com 389
    
  2. Install the ldapsearch tool to conduct subsequent tests:

    sudo yum install openldap-clients -y
    
  3. Verify basic access without specifying any credentials:

    ldapsearch -LLL -x -H ldap://ldap.hostname.com -s "base" -b "" supportedSASLMechanisms
    

    This command should return either a success response or an authentication error. Any other errors indicate that LDAP not set up correctly.

    A success response looks like the following:

    ldapsearch -LLL -x -H ldap://localhost -s "base" -b "" supportedSASLMechanisms
    dn:
    supportedSASLMechanisms: GS2-IAKERB
    supportedSASLMechanisms: GS2-KRB5
    supportedSASLMechanisms: SCRAM-SHA-1
    supportedSASLMechanisms: SCRAM-SHA-256
    supportedSASLMechanisms: GSS-SPNEGO
    supportedSASLMechanisms: GSSAPI
    supportedSASLMechanisms: DIGEST-MD5
    supportedSASLMechanisms: OTP
    supportedSASLMechanisms: NTLM
    supportedSASLMechanisms: CRAM-MD5
    

    An authentication error looks like the following:

    ldapsearch -LLL -x -W -H ldap://localhost -s "base" -b "" supportedSASLMechanisms
    Enter LDAP Password:
    ldap_bind: Invalid credentials (49)
    

    In cases where you are unable to communicate with an LDAP server, and/or the LDAP server is not set up correctly to accept LDAP requests, the error looks like the following:

    ldapsearch -LLL -x -H ldap://localhost:8090 -s "base" -b "" supportedSASLMechanisms
    ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
    
  4. Verify basic credential-based access using the same user specified in ldap.java.naming.security.principal in server.properties:

    ldapsearch -LLL -x -H ldap://ldap.hostname.com -s "base" -b "" -D CN=kafka_user,CN=Users,DC=hostname,DC=com -w 'pa55word' supportedSASLMechanisms
    

    If the value of ldap.java.naming.security.principal points to a Kerberos principal, be sure to specify the corresponding user from LDAP.

If the preceding verification tests work, then MDS can authenticate against LDAP.

When configuring Kafka to communicate over LDAPS, verify TLS connectivity to the LDAP server as follows:

# To verify the LDAP configuration
ldapsearch -LLL -x -H ldaps://ldap.hostname.com -s "base" -b "" supportedSASLMechanisms
# If using self-signed certificates from the LDAP server
LDAPTLS_CACERT=/path/to/CA.cert ldapsearch -LLL -x -H ldaps://ldap.hostname.com -s "base" -b "" supportedSASLMechanisms

When configuring Kafka to authenticate to LDAP using Kerberos, verify authentication to the LDAP server as follows:

# To verify the Kerberos configuration
kinit -k -t ./filename.keytab kafka_broker/kafka1.hostname.com@EXAMPLE.COM
ldapsearch -LLL -Y GSSAPI -H ldap://ldap.hostname.com -s "base" -b "" supportedSASLMechanisms