Use SASL/PLAIN Authentication in Confluent Platform

SASL/PLAIN overview

PLAIN, or SASL/PLAIN, is a simple username/password authentication mechanism that is typically used with TLS for encryption to implement secure authentication. Apache Kafka® supports a default implementation for SASL/PLAIN, which can be extended for production use.

The username is used as the authenticated principal, which is used in authorization (such as ACLs).

Note

PLAIN versus PLAINTEXT: Do not confuse the SASL mechanism PLAIN with the no TLS encryption option, which is called PLAINTEXT. Configuration parameters such as sasl.enabled.mechanisms or sasl.mechanism.inter.broker.protocol may be configured to use the SASL mechanism PLAIN, whereas security.inter.broker.protocol or listeners may be configured to use the no TLS encryption option, SASL_PLAINTEXT.

SASL/PLAIN should only be used with TLS as transport layer to ensure that clear passwords are not transmitted on the wire without encryption.

The default implementation of SASL/PLAIN in Confluent Platform specifies usernames and passwords in the JAAS configuration file. You can avoid storing clear passwords on disk by configuring your own callback handlers that obtain username and password from an external source using the configuration options sasl.server.callback.handler.class and sasl.client.callback.handler.class.

In production systems, external authentication servers may implement password authentication. You can plug in your own callback handlers that use external authentication servers for password verification by configuring sasl.server.callback.handler.class.

The remainder of this page shows you how to configure SASL/PLAIN for each component in Confluent Platform.

Configure Confluent Server brokers

Configure all brokers in the Kafka cluster to accept secure connections from clients. Any configuration changes made to the broker will require a rolling restart.

Enable security for Kafka brokers as described in the section below. Additionally, if you are using Confluent Control Center or Auto Data Balancer, configure your brokers for:

• Confluent Metrics Reporter

JAAS

Note

While use of separate JAAS files is supported, the recommended approach is to use step 5 in Configuration to replace the JAAS configuration described here.

Each KafkaServer/Broker uses the KafkaServer section in the JAAS file to provide SASL configuration options for the broker, including any SASL client connections made by the broker for interbroker communications. If configuring multiple listeners to use SASL, you can prefix the section name with the listener name in lowercase followed by a period (for example, sasl_ssl.KafkaServer.).

Use the Client section to authenticate a SASL connection with ZooKeeper, and to also allow brokers to set a SASL ACL on ZooKeeper nodes, which locks these nodes down so that only the brokers can modify them. You must specify the same principal name across all brokers. If you need to use a section name other than Client, specify the appropriate name (for example, -Dzookeeper.sasl.clientconfig=ZkClient) in the zookeeper.sasl.clientconfig system property.

By default, ZooKeeper uses “zookeeper” as the service name. If you want to change the service name, specify the appropriate name in the zookeeper.sasl.client.username system property (for example, -Dzookeeper.sasl.client.username=zk).

Brokers can also configure JAAS using the broker configuration property sasl.jaas.config. You must prefix the property name with the listener prefix, including the SASL mechanism: listener.name.{listenerName}.{saslMechanism}.sasl.jaas.config.

You can specify only one login module in the configuration value. If multiple mechanisms are configured on a listener, configurations must be provided for each mechanism using the listener and mechanism prefix. For example:

listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="admin" \
    password="admin-secret";
listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="admin" \
    password="admin-secret" \
    user_admin="admin-secret" \
    user_alice="alice-secret";

The properties username and password are used by the broker to initiate connections to other brokers. The set of properties user_<username> defines the passwords for all users that connect to the broker and the broker validates all client connections including those from other brokers using these properties.

If using a separate JAAS file, pass the name of the JAAS file as a JVM parameter when you start each Kafka broker:

export KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
kafka-server-start etc/kafka/server.properties

For additional options that you can pass in a JVM parameter, see Run.

If the JAAS configuration is defined at different levels, the order of precedence used is:

1. Broker configuration property (listener.name.{listenerName}.{saslMechanism}.sasl.jaas.config)
2. {listenerName}.KafkaServer section of static JAAS configuration
3. KafkaServer section of static JAAS configuration

Note that you can only configure ZooKeeper JAAS using a static JAAS configuration.

Configuration

1. Enable SASL/PLAIN mechanism in the server.properties file of every broker.

   # List of enabled mechanisms, can be more than one
   sasl.enabled.mechanisms=PLAIN
   
   # Specify one of of the SASL mechanisms
   sasl.mechanism.inter.broker.protocol=PLAIN
   

2. If you want to enable SASL for interbroker communication, add the following to the broker properties file (it defaults to PLAINTEXT). Set the protocol to:

   • SASL_SSL: if TLS/SSL encryption is enabled (TLS/SSL encryption should always be used if SASL mechanism is PLAIN)
   • SASL_PLAINTEXT: if TLS/SSL encryption is not enabled

   # Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
   security.inter.broker.protocol=SASL_SSL
   

3. Tell the Kafka brokers on which ports to listen for client and interbroker SASL connections. You must configure listeners, and optionally advertised.listeners if the value is different from listeners. Set the listener to:

   • SASL_SSL: if TLS/SSL encryption is enabled (TLS/SSL encryption should always be used if SASL mechanism is PLAIN)
   • SASL_PLAINTEXT: if TLS/SSL encryption is not enabled

   # With TLS/SSL encryption
   listeners=SASL_SSL://kafka1:9093
   advertised.listeners=SASL_SSL://localhost:9093
   
   # Without TLS/SSL encryption
   listeners=SASL_PLAINTEXT://kafka1:9093
   advertised.listeners=SASL_PLAINTEXT://localhost:9093
   

4. Configure both SASL_SSL and PLAINTEXT ports if:

   • SASL is not enabled for interbroker communication
   • Some clients connecting to the cluster do not use SASL

   Example SASL listeners with TLS/SSL encryption, mixed with PLAINTEXT listeners

   # With TLS/SSL encryption
   listeners=PLAINTEXT://kafka1:9092,SASL_SSL://kafka1:9093
   advertised.listeners=PLAINTEXT://localhost:9092,SASL_SSL://localhost:9093
   
   # Without TLS/SSL encryption
   listeners=PLAINTEXT://kafka1:9092,SASL_PLAINTEXT://kafka1:9093
   advertised.listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://localhost:9093
   

5. If you are not using a separate JAAS configuration file to configure JAAS, then configure JAAS for the Kafka broker listener as follows:

   # With TLS/SSL encryption
   listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
      username="admin" \
      password="admin-secret" \
      user_admin="admin-secret" \
      user_kafkabroker1="kafkabroker1-secret";
   
   # Without TLS/SSL encryption
   listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
      username="admin" \
      password="admin-secret" \
      user_admin="admin-secret" \
      user_kafkabroker1="kafkabroker1-secret";
   

   The properties username and password are used by the broker to initiate connections to other brokers. The set of properties user_<username> defines the passwords for all users that connect to the broker and the broker validates all client connections including those from other brokers using these properties.

Run

Following are some optional settings that you can pass in as a JVM parameter when you start each broker from the command line.

zookeeper.sasl.client

Use to enable SASL authentication to ZooKeeper.

• Type: Boolean
• Default: true
• Usage example: To pass the parameter as a JVM parameter when you start the broker, specify -Dzookeeper.sasl.client=true.

zookeeper.sasl.client.username

For SASL authentication to ZooKeeper, to change the username set the system property to use the appropriate name.

• Type: string
• Default: zookeeper
• Usage example: To pass the parameter as a JVM parameter when you start the broker, specify -Dzookeeper.sasl.client.username=zk.

zookeeper.sasl.clientconfig

Specifies the context key in the JAAS login file. This is used to change the section name for SASL authentication to ZooKeeper.

• Type: string
• Default: Client
• Usage example: To pass the parameter as a JVM parameter when you start the broker, specify -Dzookeeper.sasl.clientconfig=ZkClient.

Configure Kafka clients

Important

If you are configuring this for Schema Registry or REST Proxy, you must prefix each parameter with confluent.license. For example, sasl.mechanism becomes confluent.license.sasl.mechanism. For additional information, see Configure license clients to authenticate to Kafka.

The new Producer and Consumer clients support security for Kafka versions 0.9.0 and higher.

If you are using the Kafka Streams API, you can read on how to configure equivalent SSL and SASL parameters.

1. Configure the following properties in a client properties file client.properties.

sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
security.protocol=SASL_SSL

2. Configure the JAAS configuration property to describe how the clients like producer and consumer can connect to the Kafka Brokers. The properties username and password are used by clients to configure the user for client connections. In this example, clients connect to the broker as user kafkaclient1.

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="kafkaclient1" \
  password="kafkaclient1-secret";

Configure ZooKeeper

ZooKeeper does not support SASL/PLAIN authentication, but it does support another mechanism SASL/DIGEST-MD5.

For further details on ZooKeeper SASL authentication:

1. Client-Server mutual authentication: between the Kafka Broker (client) and ZooKeeper (server)
2. Server-Server mutual authentication: between the ZooKeeper nodes within an ensemble

Configure Kafka Connect

This section describes how to enable security for Kafka Connect. Securing Kafka Connect requires that you configure security for:

1. Kafka Connect workers: part of the Kafka Connect API, a worker is really just an advanced client, underneath the covers
2. Kafka Connect connectors: connectors may have embedded producers or consumers, so you must override the default configurations for Connect producers used with source connectors and Connect consumers used with sink connectors
3. Kafka Connect REST: Kafka Connect exposes a REST API that can be configured to use TLS/SSL using additional properties

Configure security for Kafka Connect as described in the section below. Additionally, if you are using Confluent Control Center streams monitoring for Kafka Connect, configure security for:

• Confluent Monitoring Interceptors
• Confluent Metrics Reporter

Configure all the following properties in connect-distributed.properties.

1. Configure the Connect workers to use SASL/PLAIN.

sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
security.protocol=SASL_SSL

2. Configure the JAAS configuration property to describe how Connect’s producers and consumers can connect to the Kafka Brokers. The properties username and password are used by Connect to configure the user for connections. In this example, Connect workers connect to the broker as user connect.

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

3. For the connectors to leverage security, you also have to override the default producer/consumer configuration that the worker uses. Depending on whether the connector is a source or sink connector:

• Source connector: configure the same properties adding the producer prefix.

producer.sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
producer.security.protocol=SASL_SSL
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

• Sink connector: configure the same properties adding the consumer prefix.

consumer.sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
consumer.security.protocol=SASL_SSL
consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

Configure Confluent Replicator

Confluent Replicator is a type of Kafka source connector that replicates data from a source to destination Kafka cluster. An embedded consumer inside Replicator consumes data from the source cluster, and an embedded producer inside the Kafka Connect worker produces data to the destination cluster.

Replicator version 4.0 and earlier requires a connection to ZooKeeper in the origin and destination Kafka clusters. If ZooKeeper is configured for authentication, the client configures the ZooKeeper security credentials via the global JAAS configuration setting -Djava.security.auth.login.config on the Connect workers, and the ZooKeeper security credentials in the origin and destination clusters must be the same.

To configure Confluent Replicator security, you must configure the Replicator connector as shown below and additionally you must configure:

• Kafka Connect

Configure Confluent Replicator to use SASL/PLAIN by adding these properties in the Replicator’s JSON configuration file. The JAAS configuration property defines username and password used by Replicator to configure the user for connections. In this example, Replicator connects to the broker as user replicator.

{
  "name":"replicator",
    "config":{
      ....
      "src.kafka.security.protocol" : "SASL_SSL",
      "src.kafka.sasl.mechanism" : "PLAIN",
      "src.kafka.sasl.jaas.config" : "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"replicator\" password=\"replicator-secret\";",
      ....
    }
  }
}

See also

To see an example Confluent Replicator configuration, see the SASL source authentication demo script. For demos of common security configurations see: Replicator security demos

To configure Confluent Replicator for a destination cluster with SASL/PLAIN authentication, modify the Replicator JSON configuration to include the following:

{
  "name":"replicator",
    "config":{
      ....
      "dest.kafka.security.protocol" : "SASL_SSL",
      "dest.kafka.sasl.mechanism" : "PLAIN",
      "dest.kafka.sasl.jaas.config" : "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"replicator\" password=\"replicator-secret\";",
      ....
    }
  }
}

Additionally the following properties are required in the Connect worker:

sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="replicator" password="replicator-secret";
sasl.kerberos.service.name=kafka
producer.sasl.mechanism=GSSAPI
producer.security.protocol=SASL_SSL
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="replicator" password="replicator-secret";

For more information see the general security configuration for Connect workers here.

See also

To see an example Confluent Replicator configuration, see the SASL destination authentication demo script. For demos of common security configurations see: Replicator security demos

Configure Confluent Control Center

Confluent Control Center uses Kafka Streams as a state store, so if all the Kafka brokers in the cluster backing Control Center are secured, then the Control Center application also needs to be secured.

Note

When RBAC is enabled, Control Center cannot be used in conjunction with Kerberos because Control Center cannot support any SASL mechanism other than OAUTHBEARER.

Enable security for the Control Center application as described in the section below. Additionally, configure security for the following components:

• Confluent Metrics Reporter: required on the production cluster being monitored
• Confluent Monitoring Interceptors: optional if you are using Control Center streams monitoring

1. Enable SASL/PLAIN and the security protocol for Control Center in the etc/confluent-control-center/control-center.properties file.

   confluent.controlcenter.streams.sasl.mechanism=PLAIN
   # Configure SASL_SSL if TLS/SSL encryption is enabled; otherwise configure SASL_PLAINTEXT
   confluent.controlcenter.streams.security.protocol=SASL_SSL
   

3. Configure the JAAS configuration property to describe how Control Center can connect to the Kafka Brokers. The properties username and password are used by Control Center to configure connections.

   confluent.controlcenter.streams.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
   username="confluent" \
   password="confluent-secret";
   

Configure Confluent Metrics Reporter

This section describes how to enable SASL/PLAIN for Confluent Metrics Reporter, which is used for Confluent Control Center and Auto Data Balancer.

To configure the Confluent Metrics Reporter for SASL/PLAIN, make the following configuration changes in the server.properties file in every broker in the production cluster being monitored.

1. Verify that the Confluent Metrics Reporter is enabled.

metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
confluent.metrics.reporter.bootstrap.servers=kafka1:9093

2. Enable the SASL/PLAIN mechanism for Confluent Metrics Reporter.

confluent.metrics.reporter.sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
confluent.metrics.reporter.security.protocol=SASL_SSL

Configure Confluent Monitoring Interceptors

Confluent Monitoring Interceptors are used for Confluent Control Center streams monitoring. This section describes how to enable security for Confluent Monitoring Interceptors in three places:

1. General clients
2. Kafka Connect
3. Confluent Replicator

Important

The typical use case for Confluent Monitoring Interceptors is to provide monitoring data to a separate monitoring cluster that most likely has different configurations. Interceptor configurations do not inherit configurations for the monitored component. If you wish to use configurations from the monitored component, you must add the appropriate prefix. For example, the option confluent.monitoring.interceptor.security.protocol=SSL, if being used for a producer, must be prefixed with producer. and would appear as producer.confluent.monitoring.interceptor.security.protocol=SSL.

Interceptors for General Clients

For Confluent Control Center stream monitoring to work with Kafka clients, you must configure SASL/PLAIN for the Confluent Monitoring Interceptors in each client.

1. Verify that the client has configured interceptors.

• Producer:

interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor

• Consumer:

interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor

2. Configure the SASL mechanism and security protocol for the interceptor.

confluent.monitoring.interceptor.sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
confluent.monitoring.interceptor.security.protocol=SASL_SSL

3. Configure the JAAS configuration property with a unique username and password.

confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="confluent" \
  password="confluent-secret";

Interceptors for Kafka Connect

1. For Confluent Control Center stream monitoring to work with Kafka Connect, you must configure SASL/PLAIN for the Confluent Monitoring Interceptors in Kafka Connect. Configure the Connect workers by adding these properties in connect-distributed.properties, depending on whether the connectors are sources or sinks.

• Source connector: configure the Confluent Monitoring Interceptors SASL mechanism with the producer prefix.

producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
producer.confluent.monitoring.interceptor.sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
producer.confluent.monitoring.interceptor.security.protocol=SASL_SSL

• Sink connector: configure the Confluent Monitoring Interceptors SASL mechanism with the consumer prefix.

consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
consumer.confluent.monitoring.interceptor.sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
consumer.confluent.monitoring.interceptor.security.protocol=SASL_SSL

2. Configure the JAAS configuration property with a username and password.

• Source connector: configure the Confluent Monitoring Interceptors JAAS configuration with the producer prefix.

producer.confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="confluent" \
  password="confluent-secret";

• Sink connector: configure the Confluent Monitoring Interceptors JAAS configuration with the consumer prefix.

consumer.confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="confluent" \
  password="confluent-secret";

Interceptors for Replicator

For Confluent Control Center stream monitoring to work with Replicator, you must configure SASL for the Confluent Monitoring Interceptors in the Replicator JSON configuration file. Here is an example subset of configuration properties to add.

{
  "name":"replicator",
    "config":{
      ....
      "src.consumer.group.id": "replicator",
      "src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
      "src.consumer.confluent.monitoring.interceptor.sasl.mechanism": "PLAIN",
      "src.consumer.confluent.monitoring.interceptor.security.protocol": "SASL_SSL",
      "src.consumer.confluent.monitoring.interceptor.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required \nusername=\"confluent\" \npassword=\"confluent-secret\";",
      ....
    }
  }
}

Configure Schema Registry

Schema Registry uses Kafka to persist schemas, and so it acts as a client to write data to the Kafka cluster. Therefore, if the Kafka brokers are configured for security, you should also configure Schema Registry to use security. You may also refer to the complete list of Schema Registry configuration options.

1. Here is an example subset of schema-registry.properties configuration parameters to add for SASL authentication:

kafkastore.bootstrap.servers=kafka1:9093
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
kafkastore.security.protocol=SASL_SSL
kafkastore.sasl.mechanism=PLAIN

2. Configure the JAAS configuration property to describe how Schema Registry can connect to the Kafka Brokers. The properties username and password are used by Schema Registry to configure the user for connections. In this example, Schema Registry connects to the broker as user schemaregistry.

kafkastore.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="schemaregistry" \
  password="schemaregistry-secret";

Configure REST Proxy

To secure Confluent REST Proxy for SASL you must configure security between the REST proxy and the Confluent Platform cluster.

For a complete list of all configuration options, refer to SASL Authentication.

1. Following is an example subset of kafka-rest.properties configuration parameters to add for SASL/PLAIN authentication:

client.bootstrap.servers=kafka1:9093
client.sasl.mechanism=PLAIN
# Configure SASL_SSL if TLS/SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
client.security.protocol=SASL_SSL

2. Configure the JAAS configuration property to describe how the REST Proxy can connect to the Kafka Brokers. The properties username and password are used by the REST Proxy to configure the user for connections. In this example, the REST Proxy connects to the broker as user restproxy.

client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="restproxy" \
  password="restproxy-secret";