Configuring SCRAM¶

SASL/SCRAM Overview¶

Salted Challenge Response Authentication Mechanism (SCRAM), or SASL/SCRAM, is a family of SASL mechanisms that addresses the security concerns with traditional mechanisms that perform username/password authentication like PLAIN. Apache Kafka® supports SCRAM-SHA-256 and SCRAM-SHA-512. All examples below use SCRAM-SHA-256, but you can substitute the configuration for SCRAM-SHA-512 as needed.

The SCRAM implementation in Kafka stores SCRAM credentials in ZooKeeper and is suitable for use in Kafka installations where ZooKeeper is on a private network. Because of this, you must create SCRAM credentials for users in ZooKeeper.

You typically specify the --bootstrap-server option to create and manage SCRAM credentials:

bin/kafka-configs --bootstrap-server localhost:9092 --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=alice-secret],SCRAM-SHA-512=[password=alice-secret]' --entity-type users --entity-name alice

bin/kafka-configs --bootstrap-server localhost:9092 --alter --add-config 'SCRAM-SHA-256=[password=admin-secret],SCRAM-SHA-512=[password=admin-secret]' --entity-type users --entity-name admin

However, in cases where you want Kafka brokers to authenticate to each other using SCRAM, and you want to create SCRAM credentials before the brokers are up and running, you must create SCRAM credentials for users in ZooKeeper using the --zookeeper option (you cannot use the --bootstrap-server option):

bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=alice-secret],SCRAM-SHA-512=[password=alice-secret]' --entity-type users --entity-name alice

bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 'SCRAM-SHA-256=[password=admin-secret],SCRAM-SHA-512=[password=admin-secret]' --entity-type users --entity-name admin

The default iteration count of 4096 is used if iterations are not specified. A random salt is created and the SCRAM identity consisting of salt, iterations, StoredKey and ServerKey are stored in ZooKeeper. See RFC 5802 Abstract for a brief description of SCRAM, or RFC 5802 for more details about SCRAM identity and the individual fields.

Security Considerations for SASL/SCRAM¶

The default implementation of SASL/SCRAM in Kafka stores SCRAM credentials in ZooKeeper. This is suitable for production use in installations where ZooKeeper is secure and on a private network.
For cases where you require Kafka brokers to authenticate each other using SCRAM, and you need to create SCRAM credentials before the brokers are up and running, use the --zookeeper option to create SCRAM credentials.
Kafka only supports the strong hash functions SHA-256 and SHA-512 with a minimum iteration count of 4096. Strong hash functions combined with strong passwords and high iteration counts protect against brute force attacks if ZooKeeper security is compromised.
SCRAM should be used only with TLS-encryption to prevent interception of SCRAM exchanges. This protects against dictionary or brute force attacks and against impersonation if ZooKeeper is compromised.
The default SASL/SCRAM credential store may be overridden using custom callback handlers by configuring sasl.server.callback.handler.class in installations where ZooKeeper is not secure.
For more details on security considerations, refer to RFC 5802.

The remainder of this page will show you how to configure SASL/SCRAM for each component in the Confluent Platform.

Brokers¶

Configure all brokers in the Kafka cluster to accept secure connections from clients. Any configuration changes made to the broker will require a rolling restart.

Enable security for Kafka brokers as described in the section below. Additionally, if you are using Confluent Control Center or Auto Data Balancer, configure your brokers for:

Confluent Metrics Reporter

JAAS¶

Note

While use of separate JAAS files is supported, it is not the recommended approach. Instead, use the listener configuration specified in step 5 of Configuration to replace the steps below.

First create the broker’s JAAS configuration file in each Kafka broker’s configuration directory. For this example, it is named kafka_server_jaas.conf.
In each broker’s JAAS file, configure a KafkaServer section. This configuration defines two users (admin and kafkabroker1). The properties username and password are used by the broker to initiate connections to other brokers. In this example, admin is the user for inter-broker communication.
```
KafkaServer {
   org.apache.kafka.common.security.scram.ScramLoginModule required
   username="admin"
   password="admin-secret";
};
```

Configuration¶

Enable SASL/SCRAM mechanism in the server.properties file of every broker.

# List of enabled mechanisms, can be more than one
sasl.enabled.mechanisms=SCRAM-SHA-256

# Specify one of of the SASL mechanisms
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256

If you want to enable SASL for inter-broker communication, add the following to the broker properties file (it defaults to PLAINTEXT). Set the protocol to:
- SASL_SSL: if SSL encryption is enabled (SSL encryption should always be used if SASL mechanism is PLAIN)
- SASL_PLAINTEXT: if SSL encryption is not enabled
```
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
security.inter.broker.protocol=SASL_SSL
```
Tell the Kafka brokers on which ports to listen for client and inter-broker SASL connections. You must configure listeners, and optionally advertised.listeners if the value is different from listeners. Set the listener to:
- SASL_SSL: if SSL encryption is enabled (SSL encryption should always be used if SASL mechanism is PLAIN)
- SASL_PLAINTEXT: if SSL encryption is not enabled
```
# With SSL encryption
listeners=SASL_SSL://kafka1:9093
advertised.listeners=SASL_SSL://localhost:9093

# Without SSL encryption
listeners=SASL_PLAINTEXT://kafka1:9093
advertised.listeners=SASL_PLAINTEXT://localhost:9093
```

Configure both SASL_SSL and PLAINTEXT ports if:

SASL is not enabled for inter-broker communication
Some clients connecting to the cluster do not use SASL

Example SASL listeners with SSL encryption, mixed with PLAINTEXT listeners

# With SSL encryption
listeners=PLAINTEXT://kafka1:9092,SASL_SSL://kafka1:9093
advertised.listeners=PLAINTEXT://localhost:9092,SASL_SSL://localhost:9093

# Without SSL encryption
listeners=PLAINTEXT://kafka1:9092,SASL_PLAINTEXT://kafka1:9093
advertised.listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://localhost:9093

If you are not using a separate JAAS configuration file to configure JAAS, then configure JAAS for the Kafka broker listener as follows:

listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required
   username="admin"
   password="admin-secret";

Run¶

If using a separate JAAS file, pass the name of the JAAS file as a JVM parameter when you start each Kafka broker:

export KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
bin/kafka-server-start etc/kafka/server.properties

Following are some optional settings that you can pass in as a JVM parameter when you start each broker from the command line.

zookeeper.sasl.client

Use to enable SASL authentication to ZooKeeper.

Type: Boolean
Default: true
Usage example: To pass the parameter as a JVM parameter when you start the broker, specify -Dzookeeper.sasl.client=true.

zookeeper.sasl.client.username

For SASL authentication to ZooKeeper, to change the username set the system property to use the appropriate name.

Type: string
Default: zookeeper
Usage example: To pass the parameter as a JVM parameter when you start the broker, specify -Dzookeeper.sasl.clientconfig=ZkClient.

zookeeper.sasl.clientconfig

Specifies the context key in the JAAS login file. This is used to change the section name for SASL authentication to ZooKeeper.

Type: string
Default: Client
Usage example: To pass the parameter as a JVM parameter when you start the broker, specify -Dzookeeper.sasl.clientconfig=ZkClient.

Clients¶

The new Producer and Consumer clients support security for Kafka versions 0.9.0 and higher.

If you are using the Kafka Streams API, you can read on how to configure equivalent SSL and SASL parameters.

Configure the following properties in a client properties file client.properties.

sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
security.protocol=SASL_SSL

Configure the JAAS configuration property to describe how the clients like producer and consumer can connect to the Kafka Brokers. The properties username and password are used by clients to configure the user for client connections. In this example, clients connect to the broker as user kafkaclient1.
```
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="kafkaclient1" \
  password="kafkaclient1-secret";
```

ZooKeeper¶

ZooKeeper does not support SASL/SCRAM authentication, but it does support another mechanism SASL/DIGEST-MD5.

For further details on ZooKeeper SASL authentication:

Client-Server mutual authentication: between the Kafka Broker (client) and ZooKeeper (server)
Server-Server mutual authentication: between the ZooKeeper nodes within an ensemble

Kafka Connect¶

This section describes how to enable security for Kafka Connect. Securing Kafka Connect requires that you configure security for:

Kafka Connect workers: part of the Kafka Connect API, a worker is really just an advanced client, underneath the covers
Kafka Connect connectors: connectors may have embedded producers or consumers, so you must override the default configurations for Connect producers used with source connectors and Connect consumers used with sink connectors
Kafka Connect REST: Kafka Connect exposes a REST API that can be configured to use SSL using additional properties

Configure security for Kafka Connect as described in the section below. Additionally, if you are using Confluent Control Center streams monitoring for Kafka Connect, configure security for:

Confluent Monitoring Interceptors
Confluent Metrics Reporter

Configure all the following properties in connect-distributed.properties.

Configure the Connect workers to use SASL/SCRAM.

sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
security.protocol=SASL_SSL

Configure the JAAS configuration property to describe how Connect’s producers and consumers can connect to the Kafka Brokers. The properties username and password are used by Connect to configure the user for connections. In this example, Connect workers connect to the broker as user connect.
```
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="connect" \
  password="connect-secret";
```

For the connectors to leverage security, you also have to override the default producer/consumer configuration that the worker uses. Depending on whether the connector is a source or sink connector:

Source connector: configure the same properties adding the producer prefix.

producer.sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
producer.security.protocol=SASL_SSL
producer.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="connect" \
  password="connect-secret";

Sink connector: configure the same properties adding the consumer prefix.

consumer.sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
consumer.security.protocol=SASL_SSL
consumer.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="connect" \
  password="connect-secret";

Confluent Replicator¶

Confluent Replicator is a type of Kafka source connector that replicates data from a source to destination Kafka cluster. An embedded consumer inside Replicator consumes data from the source cluster, and an embedded producer inside the Kafka Connect worker produces data to the destination cluster.

Replicator version 4.0 and earlier requires a connection to ZooKeeper in the origin and destination Kafka clusters. If ZooKeeper is configured for authentication, the client configures the ZooKeeper security credentials via the global JAAS configuration setting -Djava.security.auth.login.config on the Connect workers, and the ZooKeeper security credentials in the origin and destination clusters must be the same.

To configure Confluent Replicator security, you must configure the Replicator connector as shown below and additionally you must configure:

Kafka Connect

Configure Confluent Replicator to use SASL/SCRAM by adding these properties in the Replicator’s JSON configuration file. The JAAS configuration property defines username and password used by Replicator to configure the user for connections. In this example, Replicator connects to the broker as user replicator.

{
  "name":"replicator",
    "config":{
      ....
      "src.kafka.security.protocol" : "SASL_SSL",
      "src.kafka.sasl.mechanism" : "SCRAM-SHA-256",
      "src.kafka.sasl.jaas.config" : "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"replicator\" password=\"replicator-secret\";",
      ....
    }
  }
}

Confluent Control Center¶

Confluent Control Center uses Kafka Streams as a state store, so if all the Kafka brokers in the cluster backing Control Center are secured, then the Control Center application also needs to be secured.

Note

When RBAC is enabled, Control Center cannot be used in conjunction with Kerberos because Control Center cannot support any SASL mechanism other than OAUTHBEARER.

Enable security for the Control Center application as described in the section below. Additionally, configure security for the following components:

Confluent Metrics Reporter: required on the production cluster being monitored
Confluent Monitoring Interceptors: optional if you are using Control Center streams monitoring

Enable SASL/SCRAM and the security protocol for Control Center in the etc/confluent-control-center/control-center.properties file.

confluent.controlcenter.streams.sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
confluent.controlcenter.streams.security.protocol=SASL_SSL

Configure the JAAS configuration property to describe how Control Center can connect to the Kafka Brokers. The properties username and password are used by Control Center to configure connections.
```
confluent.controlcenter.streams.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="confluent" \
  password="confluent-secret";
```

Confluent Metrics Reporter¶

This section describes how to enable SASL/SCRAM for Confluent Metrics Reporter, which is used for Confluent Control Center and Auto Data Balancer.

To configure the Confluent Metrics Reporter for SASL/SCRAM, make the following configuration changes in the server.properties file in every broker in the production cluster being monitored.

Verify that the Confluent Metrics Reporter is enabled.

metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
confluent.metrics.reporter.bootstrap.servers=kafka1:9093

Enable the SASL/SCRAM mechanism for Confluent Metrics Reporter.

confluent.metrics.reporter.sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
confluent.metrics.reporter.security.protocol=SASL_SSL

Confluent Monitoring Interceptors¶

Confluent Monitoring Interceptors are used for Confluent Control Center streams monitoring. This section describes how to enable security for Confluent Monitoring Interceptors in three places:

General clients
Kafka Connect
Confluent Replicator

Important

The typical use case for Confluent Monitoring Interceptors is to provide monitoring data to a separate monitoring cluster that most likely has different configurations. Interceptor configurations do not inherit configurations for the monitored component. If you wish to use configurations from the monitored component, you must add the appropriate prefix. For example, the option confluent.monitoring.interceptor.security.protocol=SSL, if being used for a producer, must be prefixed with producer. and would appear as producer.confluent.monitoring.interceptor.security.protocol=SSL.

Interceptors for General Clients¶

For Confluent Control Center stream monitoring to work with Kafka clients, you must configure SASL/SCRAM for the Confluent Monitoring Interceptors in each client.

Verify that the client has configured interceptors.

Producer:

interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor

Consumer:

interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor

Configure the SASL mechanism and security protocol for the interceptor.

confluent.monitoring.interceptor.sasl.mechanism=SCRAM
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
confluent.monitoring.interceptor.security.protocol=SASL_SSL

Configure the JAAS configuration property with a unique username and password.

confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="confluent" \
  password="confluent-secret";

Interceptors for Kafka Connect¶

For Confluent Control Center stream monitoring to work with Kafka Connect, you must configure SASL/SCRAM for the Confluent Monitoring Interceptors in Kafka Connect. Configure the Connect workers by adding these properties in connect-distributed.properties, depending on whether the connectors are sources or sinks.

Source connector: configure the Confluent Monitoring Interceptors SASL mechanism with the producer prefix.

producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
producer.confluent.monitoring.interceptor.sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
producer.confluent.monitoring.interceptor.security.protocol=SASL_SSL

Sink connector: configure the Confluent Monitoring Interceptors SASL mechanism with the consumer prefix.

consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
consumer.confluent.monitoring.interceptor.sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
consumer.confluent.monitoring.interceptor.security.protocol=SASL_SSL

Configure the JAAS configuration property with a username and password.

Source connector: configure the Confluent Monitoring Interceptors JAAS configuration with the producer prefix.

producer.confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="confluent" \
  password="confluent-secret";

Sink connector: configure the Confluent Monitoring Interceptors JAAS configuration with the consumer prefix.

consumer.confluent.monitoring.interceptor.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="confluent" \
  password="confluent-secret";

Interceptors for Replicator¶

For Confluent Control Center stream monitoring to work with Replicator, you must configure SASL for the Confluent Monitoring Interceptors in the Replicator JSON configuration file. Here is an example subset of configuration properties to add.

{
  "name":"replicator",
    "config":{
      ....
      "src.consumer.group.id": "replicator",
      "src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
      "src.consumer.confluent.monitoring.interceptor.sasl.mechanism": "SCRAM",
      "src.consumer.confluent.monitoring.interceptor.security.protocol": "SASL_SSL",
      "src.consumer.confluent.monitoring.interceptor.sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required \nusername=\"confluent\" \npassword=\"confluent-secret\";",
      ....
    }
  }
}

Schema Registry¶

Important

If you are configuring this for Schema Registry or REST Proxy, you must prefix each parameter with confluent.license. For example, sasl.mechanism becomes confluent.license.sasl.mechanism.

Schema Registry uses Kafka to persist schemas, and so it acts as a client to write data to the Kafka cluster. Therefore, if the Kafka brokers are configured for security, you should also configure Schema Registry to use security. You may also refer to the complete list of Schema Registry configuration options.

Here is an example subset of schema-registry.properties configuration parameters to add for SASL authentication:

kafkastore.bootstrap.servers=kafka1:9093
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
kafkastore.security.protocol=SASL_SSL
kafkastore.sasl.mechanism=SCRAM-SHA-256

Configure the JAAS configuration property to describe how Schema Registry can connect to the Kafka Brokers. The properties username and password are used by Schema Registry to configure the user for connections. In this example, Schema Registry connects to the broker as user schemaregistry.
```
kafkastore.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="schemaregistry" \
  password="schemaregistry-secret";
```

REST Proxy¶

Important

If you are configuring this for Schema Registry or REST Proxy, you must prefix each parameter with confluent.license. For example, sasl.mechanism becomes confluent.license.sasl.mechanism.

Securing Confluent REST Proxy for SASL requires that you configure security between the REST proxy and the Kafka cluster.

For a complete list of all configuration options, refer to SASL Authentication.

Configure the SASL/SCRAM mechanism in kafka-rest.properties.

Note

Make sure the bootstrap.servers configuration is set with SASL_PLAINTEXT://host:port (or SASL_SSL://host:port) endpoints, or you’ll accidentally open an SASL connection to a non-SASL port. For more details, see bootstrap.servers in REST Proxy Configuration Options.
```
bootstrap.servers=SASL_SSL://kafka1:9093
client.sasl.mechanism=SCRAM-SHA-256
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
client.security.protocol=SASL_SSL
```
Configure the JAAS configuration property to describe how the REST Proxy can connect to the Kafka Brokers. The properties username and password are used by the REST Proxy to configure the user for connections. In this example, the REST Proxy connects to the broker as user restproxy.
```
client.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="restproxy" \
  password="restproxy-secret";
```