Use TLS Authentication in Confluent Platform¶

TLS overview¶

Confluent Platform supports Transport Layer Security (TLS) encryption based on OpenSSL, an open source cryptography toolkit that provides an implementation of the Transport Layer Security (TLS) and Secure Socket Layer (SSL) protocols With TLS authentication, the server authenticates the client (also called “two-way authentication”).

Note

You can learn more about securing your Kafka deployment in the free course, Apache Kafka Security.

Because TLS authentication requires TLS encryption, this page shows you how to configure both at the same time and is a superset of configurations required just for TLS encryption.

By default, Apache Kafka® communicates in PLAINTEXT, which means that all data is sent in plain text (unencrypted). To encrypt communication, you should configure all the Confluent Platform components in your deployment to use TLS encryption.

Confluent Platform supports Transport Layer Security (TLS) encryption based on OpenSSL, an open source cryptography toolkit that provides an implementation of the Transport Layer Security (TLS) and Secure Socket Layer (SSL) protocols With TLS authentication, the server authenticates the client (also called mutual authentication (mTLS)).

Secure Sockets Layer (SSL) was the predecessor of Transport Layer Security (TLS), and has been deprecated since June 2015. For TLS configurations, the properties often still refer to SSL.

You can configure TLS for encryption, but you can also configure TLS for authentication. You can configure just TLS encryption (by default, TLS encryption includes certificate authentication of the server) and independently choose a separate mechanism for client authentication (for example, TLS, SASL). Technically speaking, TLS encryption already enables one-way authentication in which the client authenticates the server certificate. In this topic, “TLS authentication” refers to the two-way authentication, where the broker also authenticates the client certificate.

Enabling TLS may have a performance impact due to encryption overhead.

TLS uses private-key/certificate pairs, which are used during the TLS handshake process.

Each broker needs its own private-key/certificate pair, and the client uses the certificate to authenticate the broker.
Each logical client needs a private-key/certificate pair if client authentication is enabled, and the broker uses the certificate to authenticate the client.

You can configure each broker and logical client with a truststore, which is used to determine which certificates (broker or logical client identities) to trust (authenticate). You can configure the truststore in many ways. Consider the following two examples:

The truststore contains one or many certificates: the broker or logical client will trust any certificate listed in the truststore.
The truststore contains a Certificate Authority (CA): the broker or logical client will trust any certificate that was signed by the CA in the truststore.

Using the CA method is more convenient, because adding a new broker or client doesn’t require a change to the truststore. The CA method is outlined in this diagram.

However, with the CA method, Kafka does not conveniently support blocking authentication for individual brokers or clients that were previously trusted using this mechanism (certificate revocation is typically done using Certificate Revocation Lists or the Online Certificate Status Protocol), so you would have to rely on authorization to block access.

In contrast, if you use one or many certificates, blocking authentication is achieved by removing the broker or client’s certificate from the truststore.

Creating TLS keys and certificates¶

Refer to the Enable Security for a ZooKeeper-Based Cluster in Confluent Platform, which describes how to create TLS keys and certificates.

Brokers¶

Configure all brokers in the Kafka cluster to accept secure connections from clients. Any configuration changes made to the broker will require a rolling restart.

Enable security for Kafka brokers as described in the section below. Additionally, if you are using Confluent Control Center or Auto Data Balancer, configure your brokers for:

Confluent Metrics Reporter

Note

For details on all required and optional broker configuration properties, see Kafka Broker and Controller Configuration Reference for Confluent Platform.

Configure the truststore, keystore, and password in the server.properties file of every broker. Because this stores passwords directly in the broker configuration file, it is important to restrict access to these files using file system permissions.
```
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=test1234
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234
```
Note that ssl.truststore.password is technically optional, but strongly recommended. If a password is not set, access to the truststore is still available, but integrity checking is disabled.
If you want to enable TLS for interbroker communication, add the following to the broker properties file, which defaults to PLAINTEXT:
```
security.inter.broker.protocol=SSL
```
Configure the ports for the Apache Kafka® brokers to listen for client and interbroker TLS (SSL) connections. You should configure listeners, and optionally, advertised.listeners if the value is different from listeners.
```
listeners=SSL://kafka1:9093
advertised.listeners=SSL://localhost:9093
```
Configure both TLS (SSL) ports and PLAINTEXT ports if:
- TLS is not enabled for interbroker communication
- Some clients connecting to the Confluent Platform cluster do not use TLS
```
listeners=PLAINTEXT://kafka1:9092,SSL://kafka1:9093
advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
```
Note that advertised.host.name and advertised.port configure a single PLAINTEXT port and are incompatible with secure protocols. Use advertised.listeners instead.
To enable the broker to authenticate clients (two-way authentication), you must configure all the brokers for client authentication. Configure this to use required rather than requested because misconfigured clients can still connect successfully and this provides a false sense of security.
```
ssl.client.auth=required
```
Note

If you specify ssl.client.auth=required, client authentication fails if valid client certificates are not provided. SASL listeners can be enabled in parallel to mTLS if you have defined SASL listeners with the following listener prefix:
```
listener.name.<saslListenerName>.ssl.client.auth
```
For details, see KIP-684.

Optional settings¶

ssl.endpoint.identification.algorithm: The endpoint identification algorithm used by clients to validate server host name. The default value is https. Clients including client connections created by the broker for interbroker communication verify that the broker host name matches the host name in the broker’s certificate. Disable server host name verification by setting ssl.endpoint.identification.algorithm to an empty string.

Type: string
Default: https
Importance: medium

ssl.cipher.suites: A cipher suite is a named combination of authentication, encryption, MAC, and key exchange algorithm used to negotiate the security settings for a network connection (using the TLS network protocol).

Type: list
Default: null (by default, all supported cipher suites are enabled)
Importance: medium

ssl.enabled.protocols: The comma-separated list of protocols enabled for TLS connections. The default value is TLSv1.2,TLSv1.3 when running with Java 11 or later, TLSv1.2 otherwise. With the default value for Java 11 (TLSv1.2,TLSv1.3), Kafka clients and brokers prefer TLSv1.3 if both support it, and falls back to TLSv1.2 otherwise (assuming both support at least TLSv1.2).

Type: list
Default: TLSv1.2, TLSv1.3
Importance: medium

ssl.truststore.type: The file format of the truststore file.

Type: string
Default: JKS
Importance: medium

Due to import regulations in some countries, the Oracle implementation limits the strength of cryptographic algorithms available by default. If stronger algorithms are needed (for example, AES with 256-bit keys), the JCE Unlimited Strength Jurisdiction Policy Files must be obtained and installed in the JDK/JRE. See the JCA Providers Documentation for more information.

Clients¶

Important

If you are configuring this for Schema Registry or REST Proxy, you must prefix each parameter with confluent.license. For example, sasl.mechanism becomes confluent.license.sasl.mechanism. For additional information, see Configure license clients to authenticate to Kafka.

The new Producer and Consumer clients support security for Kafka versions 0.9.0 and higher.

If you are using the Kafka Streams API, you can read on how to configure equivalent SSL and SASL parameters.

In the following configuration example, the underlying assumption is that client authentication is required by the broker so that you can store it in a client properties file client-ssl.properties. Because this stores passwords directly in the broker configuration file, it is important to restrict access to these files using file system permissions.

bootstrap.servers=kafka1:9093
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password=test1234
ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234

Note that ssl.truststore.password is technically optional, but strongly recommended. If a password is not set, access to the truststore is still available, but integrity checking is disabled.

The following examples use kafka-console-producer and kafka-console-consumer, and pass in the client-ssl.properties defined above:

kafka-console-producer --bootstrap-server kafka1:9093 --topic test --producer.config client-ssl.properties
kafka-console-consumer --bootstrap-server kafka1:9093 --topic test --consumer.config client-ssl.properties --from-beginning

Optional settings¶

ssl.endpoint.identification.algorithm

The endpoint identification algorithm used by clients to validate server host name. The default value is https. Clients including client connections created by the broker for interbroker communication verify that the broker host name matches the host name in the broker’s certificate. Disable server host name verification by setting ssl.endpoint.identification.algorithm to an empty string.

Type: string
Default: https
Importance: medium

ssl.provider

The name of the security provider used for TLS connections. Default value is the default security provider of the JVM.

Type: string
Default: null
Importance: medium

ssl.cipher.suites

A cipher suite is a named combination of authentication, encryption, MAC, and key exchange algorithm used to negotiate the security settings for a network connection (using the TLS network protocol).

Type: list
Default: null (by default, all supported cipher suites are enabled)
Importance: medium

ssl.enabled.protocols

The comma-separated list of protocols enabled for TLS connections. The default value is TLSv1.2,TLSv1.3 when running with Java 11 or later, TLSv1.2 otherwise. With the default value for Java 11 (TLSv1.2,TLSv1.3), Kafka clients and brokers prefer TLSv1.3 if both support it, and falls back to TLSv1.2 otherwise (assuming both support at least TLSv1.2).

Type: list
Default: TLSv1.2,TLSv1.3
Importance: medium

ssl.truststore.type

The file format of the truststore file.

Type: string
Default: JKS
Importance: medium

ZooKeeper¶

Starting in Confluent Platform version 5.5.0, the version of ZooKeeper bundled with Confluent Platform supports TLS. For details, refer to Add Security to Running Clusters in Confluent Platform.

Important

As of Confluent Platform 7.5, ZooKeeper is deprecated for new deployments. Confluent recommends KRaft mode for new deployments. For more information, see KRaft Overview for Confluent Platform.

Kafka Connect¶

This section describes how to enable security for Kafka Connect. Securing Kafka Connect requires that you configure security for:

Kafka Connect workers: part of the Kafka Connect API, a worker is really just an advanced client, underneath the covers
Kafka Connect connectors: connectors may have embedded producers or consumers, so you must override the default configurations for Connect producers used with source connectors and Connect consumers used with sink connectors
Kafka Connect REST: Kafka Connect exposes a REST API that can be configured to use TLS/SSL using additional properties

Configure security for Kafka Connect as described in the section below. Additionally, if you are using Confluent Control Center streams monitoring for Kafka Connect, configure security for:

Confluent Monitoring Interceptors
Confluent Metrics Reporter

Configure the top-level settings in the Connect workers to use TLS by adding these properties in connect-distributed.properties. These top-level settings are used by the Connect worker for group coordination and to read and write to the internal topics that are used to track the cluster’s state (for example, configs and offsets). The assumption here is that client authentication is required by the brokers.

bootstrap.servers=kafka1:9093
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password=test1234
ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234

Connect workers manage the producers used by source connectors and the consumers used by sink connectors. So, for the connectors to leverage security, must also override the default producer/consumer configuration that the worker uses. The assumption here is that client authentication is required by the brokers.

For source connectors: configure the same properties adding the producer prefix.

producer.bootstrap.servers=kafka1:9093
producer.security.protocol=SSL
producer.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
producer.ssl.truststore.password=test1234
producer.ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
producer.ssl.keystore.password=test1234
producer.ssl.key.password=test1234

For sink connectors: configure the same properties adding the consumer prefix.

consumer.bootstrap.servers=kafka1:9093
consumer.security.protocol=SSL
consumer.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
consumer.ssl.truststore.password=test1234
consumer.ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
consumer.ssl.keystore.password=test1234
consumer.ssl.key.password=test1234

Important

Updating the certificate authority (CA)

When you update the certificate authority (CA), the Connect workers must be restarted.

You can pass the CA at the JVM level as shown here:

KAFKA_OPTS="-Djavax.net.ssl.trustStore=${CERTS_PATH}/truststore.jks \
-Djavax.net.ssl.trustStorePassword=$TRUSTSTORE_PASSWORD \
-Djavax.net.ssl.keyStore=${CERTS_PATH}/keystore.jks \
-Djavax.net.ssl.keyStorePassword=$KEYSTOR

For more information, see Kafka Connect Security Basics.

Confluent Replicator¶

Confluent Replicator is a type of Kafka source connector that replicates data from a source to destination Kafka cluster. An embedded consumer inside Replicator consumes data from the source cluster, and an embedded producer inside the Kafka Connect worker produces data to the destination cluster.

Replicator version 4.0 and earlier requires a connection to ZooKeeper in the origin and destination Kafka clusters. If ZooKeeper is configured for authentication, the client configures the ZooKeeper security credentials via the global JAAS configuration setting -Djava.security.auth.login.config on the Connect workers, and the ZooKeeper security credentials in the origin and destination clusters must be the same.

To configure Confluent Replicator security, you must configure the Replicator connector as shown below and additionally you must configure:

Kafka Connect

To add TLS to the Confluent Replicator embedded consumer, modify the Replicator JSON properties file.

This example is a subset of configuration properties to add for TLS encryption and authentication. The assumption here is that client authentication is required by the brokers.

{
  "name":"replicator",
    "config":{
      ....
      "src.kafka.ssl.truststore.location":"/etc/kafka/secrets/kafka.connect.truststore.jks",
      "src.kafka.ssl.truststore.password":"confluent",
      "src.kafka.ssl.keystore.location":"/etc/kafka/secrets/kafka.connect.keystore.jks",
      "src.kafka.ssl.keystore.password":"confluent",
      "src.kafka.ssl.key.password":"confluent",
      "src.kafka.security.protocol":"SSL"
      ....
    }
  }
}

Confluent Control Center¶

You can configure TLS for Control Center so access is secured through HTTPS.

In addition, Control Center uses Kafka Streams as a state store, so if all the Confluent Server brokers in the Confluent Platform cluster backing Confluent Control Center are secured, then Confluent Control Center also needs to be secured.

Also, since the Control Center acts as a proxy server for other components, you can configure TLS for Control Center to secure its communication with other secured Confluent Platform components.

Enable TLS for Confluent Control Center in the etc/confluent-control-center/control-center.properties file. The assumption here is that client authentication is required by the brokers.

For details on how to enable TLS for Control Center as a server or a proxy server, see Configure TLS for Control Center on Confluent Platform.

Confluent Metrics Reporter¶

This section describes how to enable TLS encryption and authentication for Confluent Metrics Reporter, which is used for Confluent Control Center and Auto Data Balancer.

To add TLS for the Confluent Metrics Reporter, add the following to server.properties on the brokers in the Confluent Platform cluster being monitored. The assumption here is that client authentication is required by the brokers.

confluent.metrics.reporter.bootstrap.servers=kafka1:9093
confluent.metrics.reporter.security.protocol=SSL
confluent.metrics.reporter.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
confluent.metrics.reporter.ssl.truststore.password=test1234
confluent.metrics.reporter.ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
confluent.metrics.reporter.ssl.keystore.password=test1234
confluent.metrics.reporter.ssl.key.password=test1234

Confluent Monitoring Interceptors¶

Confluent Monitoring Interceptors are used for Confluent Control Center streams monitoring. This section describes how to enable security for Confluent Monitoring Interceptors in three places:

General clients
Kafka Connect
Confluent Replicator

Important

The typical use case for Confluent Monitoring Interceptors is to provide monitoring data to a separate monitoring cluster that most likely has different configurations. Interceptor configurations do not inherit configurations for the monitored component. If you wish to use configurations from the monitored component, you must add the appropriate prefix. For example, the option confluent.monitoring.interceptor.security.protocol=SSL, if being used for a producer, must be prefixed with producer. and would appear as producer.confluent.monitoring.interceptor.security.protocol=SSL.

Interceptors for general clients¶

For Confluent Control Center stream monitoring to work with Kafka clients, you must configure TLS encryption and authentication for the Confluent Monitoring Interceptors in each client.

Verify that the client has configured interceptors.

Producer:

interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor

Consumer:

interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor

Configure TLS encryption and authentication for the interceptor. The assumption here is that client authentication is required by the brokers.

confluent.monitoring.interceptor.bootstrap.servers=kafka1:9093
confluent.monitoring.interceptor.security.protocol=SSL
confluent.monitoring.interceptor.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
confluent.monitoring.interceptor.ssl.truststore.password=test1234
confluent.monitoring.interceptor.ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
confluent.monitoring.interceptor.ssl.keystore.password=test1234
confluent.monitoring.interceptor.ssl.key.password=test1234

Interceptors for Kafka Connect¶

For Confluent Control Center stream monitoring to work with Kafka Connect, you must configure TLS for the Confluent Monitoring Interceptors in Kafka Connect. The assumption here is that client authentication is required by the brokers. Configure the Connect workers by adding these properties in connect-distributed.properties, depending on whether the connectors are sources or sinks.

Source connector: configure the Confluent Monitoring Interceptors for TLS encryption and authentication with the producer prefix.

producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
producer.confluent.monitoring.interceptor.bootstrap.servers=kafka1:9093
producer.confluent.monitoring.interceptor.security.protocol=SSL
producer.confluent.monitoring.interceptor.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
producer.confluent.monitoring.interceptor.ssl.truststore.password=test1234
producer.confluent.monitoring.interceptor.ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
producer.confluent.monitoring.interceptor.ssl.keystore.password=test1234
producer.confluent.monitoring.interceptor.ssl.key.password=test1234

Sink connector: configure the Confluent Monitoring Interceptors for TLS encryption and authentication with the consumer prefix.

consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
consumer.confluent.monitoring.interceptor.bootstrap.servers=kafka1:9093
consumer.confluent.monitoring.interceptor.security.protocol=SSL
consumer.confluent.monitoring.interceptor.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
consumer.confluent.monitoring.interceptor.ssl.truststore.password=test1234
consumer.confluent.monitoring.interceptor.ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
consumer.confluent.monitoring.interceptor.ssl.keystore.password=test1234
consumer.confluent.monitoring.interceptor.ssl.key.password=test1234

Interceptors for Replicator¶

For Confluent Control Center stream monitoring to work with Replicator, you must configure TLS for the Confluent Monitoring Interceptors in the Replicator JSON configuration file. Here is an example subset of configuration properties to add for TLS encryption and authentication:

{
  "name":"replicator",
    "config":{
      ....
      "src.consumer.group.id": "replicator",
      "src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
      "src.consumer.confluent.monitoring.interceptor.security.protocol": "SSL",
      "src.consumer.confluent.monitoring.interceptor.bootstrap.servers": "kafka1:9093",
      "src.consumer.confluent.monitoring.interceptor.ssl.truststore.location": "/var/private/ssl/kafka.client.truststore.jks",
      "src.consumer.confluent.monitoring.interceptor.ssl.truststore.password": "test1234",
      "src.consumer.confluent.monitoring.interceptor.ssl.keystore.location": "/var/private/ssl/kafka.client.keystore.jks",
      "src.consumer.confluent.monitoring.interceptor.ssl.keystore.password": "test1234",
      "src.consumer.confluent.monitoring.interceptor.ssl.key.password": "test1234",
      ....
    }
  }
}

Enable TLS in a Self-Balancing cluster¶

To enable TLS encryption in a Self-Balancing cluster, add the following to the server.properties file on the brokers in the Confluent Platform cluster.

confluent.rebalancer.metrics.security.protocol=SSL
confluent.rebalancer.metrics.ssl.truststore.location=/etc/kafka/secrets/kafka.client.truststore.jks
confluent.rebalancer.metrics.ssl.truststore.password=confluent
confluent.rebalancer.metrics.ssl.keystore.location=/etc/kafka/secrets/kafka.client.keystore.jks
confluent.rebalancer.metrics.ssl.keystore.password=confluent
confluent.rebalancer.metrics.ssl.key.password=confluent

Schema Registry¶

Schema Registry uses Kafka to persist schemas, and so it acts as a client to write data to the Kafka cluster. Therefore, if the Kafka brokers are configured for security, you should also configure Schema Registry to use security. You may also refer to the complete list of Schema Registry configuration options.

The following is an example subset of schema-registry.properties configuration parameters to add for TLS encryption and authentication. The assumption here is that client authentication is required by the brokers.

kafkastore.bootstrap.servers=SSL://kafka1:9093
kafkastore.security.protocol=SSL
kafkastore.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
kafkastore.ssl.truststore.password=test1234
kafkastore.ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
kafkastore.ssl.keystore.password=test1234
kafkastore.ssl.key.password=test1234

REST Proxy¶

Securing Confluent REST Proxy with TLS encryption and authentication requires that you configure security between:

REST clients and the REST Proxy (HTTPS)
REST proxy and the Confluent Platform cluster

Also, refer to the complete list of REST Proxy configuration options.

Configure HTTPS between REST clients and the REST Proxy. The following is an example subset of kafka-rest.properties configuration parameters to configure HTTPS.

ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password=test1234
ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234

Configure TLS encryption and authentication between REST proxy and the Confluent Platform cluster. The following is an example subset of kafka-rest.properties configuration parameters to add for TLS encryption and authentication. The assumption here is that client authentication is required by the brokers.

client.bootstrap.servers=kafka1:9093
client.security.protocol=SSL
client.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
client.ssl.truststore.password=test1234
client.ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
client.ssl.keystore.password=test1234
client.ssl.key.password=test1234

TLS Logging¶

Enable TLS debug logging at the JVM level by starting the Kafka broker and/or clients with the javax.net.debug system property. For example:

export KAFKA_OPTS=-Djavax.net.debug=all
kafka-server-start etc/kafka/server.properties

Tip

These instructions are based on the assumption that you are installing Confluent Platform by using ZIP or TAR archives. For more information, see Install Confluent Platform On-Premises.

Once you start the broker you should be able to see in the server.log:

with addresses: PLAINTEXT -> EndPoint(192.168.64.1,9092,PLAINTEXT),SSL -> EndPoint(192.168.64.1,9093,SSL)

To verify if the server’s keystore and truststore are setup correctly you can run the following command:

openssl s_client -debug -connect localhost:9093 -tls1

Note: TLSv1 should be listed under ssl.enabled.protocols.

In the output of this command you should see the server’s certificate:

You can find more details on this in the Oracle documentation on debugging SSL/TLS connections.

If the certificate does not show up with the openssl command, or if there are any other error messages, then your keys or certificates are not setup correctly. Review your configurations.