Secure ZooKeeper in Confluent Platform¶

ZooKeeper authentication overview¶

As of version 3.5.x, ZooKeeper supports mutual TLS (mTLS) authentication. As of version 2.5, Kafka supports authenticating to ZooKeeper with SASL and mTLS–either individually or together. See KIP-515 for details.

When using mTLS alone, every broker and CLI tool (such as the ZooKeeper security migration tool, ZkSecurityMigrator) must identify itself using the same Distinguished Name (DN). The DN is included in the ZooKeeper ACL, so ZooKeeper will only authorize that DN; therefore all connections to ZooKeeper must provide the DN. You can modify what gets put into the ACL from the DN, but it involves writing and deploying a custom ZooKeeper authentication provider. For details, see mTLS authentication. Each CA certificate should use the same DN, and also specify a different Subject Alternative Name (SAN) to ensure that ZooKeeper hostname verification of brokers and any CLI tools will succeed.

When using SASL and mTLS authentication simultaneously with ZooKeeper, the SASL identity and either the DN that created the znode (the creating broker’s CA certificate) or the DN of the security migration tool (if migration was performed after the znode was created) are included in ACLs. Hence, all brokers and CLI tools will be authorized even if they all use different DNs because they all use the same SASL identity. When using mTLS authentication only, all DNs must match, and SANs become critical in the absence of writing and deploying a custom ZooKeeper authentication provider.

Important

As of Confluent Platform 7.5, ZooKeeper is deprecated for new deployments. Confluent recommends KRaft mode for new deployments. For more information, see KRaft Overview for Confluent Platform.

You can enable security in ZooKeeper by using the examples below.

Note

When authenticating brokers with ZooKeeper, set zookeeper.set.acl=true for all brokers. If you accept the default (zookeeper.set.acl=false), then no ACLs are created and the information in ZooKeeper will be writeable by everyone.

SASL authentication¶

This section describes how to enable and use SASL authentication with ZooKeeper. For details about adding TLS encryption so that SASL credentials are not transmitted in the clear, refer to Encrypting communication to ZooKeeper with TLS.

Enable ZooKeeper authentication with SASL¶

To enable ZooKeeper authentication with SASL add the following to zookeeper.properties:

authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

SASL with Digest-MD5¶

Here is an example of a ZooKeeper node JAAS file:

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_super="adminsecret"
       user_bob="bobsecret";
};

Here is an example of a ZooKeeper client JAAS file, including brokers and admin scripts like kafka-topics:

Client {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       username="bob"
       password="bobsecret";
};

If your Confluent Server broker already has a JAAS file, this section must be added to it.

SASL with Kerberos¶

Here is an example of ZooKeeper node JAAS file:

Server {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="<path-to-server-keytab>"
    storeKey=true
    useTicketCache=false
    principal="zookeeper/yourzkhostname@EXAMPLE.COM";
};

Here is an example of a ZooKeeper client JAAS file, including brokers and admin scripts like kafka-topics:

Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    keyTab="<path-to-client-keytab>"
    principal="kafka/kafka1.hostname.com@EXAMPLE.COM";
};

Note

Before starting ZooKeeper, check the JAAS syntax and keytab permissions. The most common errors that prevent the server from starting are JAAS syntax errors or permissions set incorrectly on the keytab file. Refer to Encrypting communication to ZooKeeper with TLS for details.

mTLS authentication¶

You can enable ZooKeeper mTLS authentication with or without SASL authentication.

All clients that connect to ZooKeeper must share an identity; every connection has zero or more identities associated with it. Two identities are possible, for example, if both mTLS and SASL are enabled and the client successfully authenticates with both. If you use mTLS with SASL, then mTLS doesn’t have to be the source of the common identity–the SASL identity can be the same for everyone.

When enabling mTLS without SASL authentication, every broker and/or CLI tool (such as the ZooKeeper security migration tool, ZkSecurityMigrator) must identify itself using the same Distinguished Name (DN). By default, the full DN is included in the ZooKeeper ACL, and ZooKeeper only authorizes what is in the ACL, so all connections to ZooKeeper should provide that DN. However, ZooKeeper cannot use this single DN to map to multiple hosts from which requests originate. Therefore, each CA certificate should include a subject alternative name (SAN). If there is no SAN, hostname verification of brokers and/or any CLI tools will fail and ZooKeeper will reject the connection.

To illustrate the DN and SAN requirement, consider the scenario where you browse the web site https://meeting.bigdata.us. The CA certificate for the site must include the DN CN=meeting.bigdata.us (or a wildcard certificate with CN=*.bigdata.us); if it does not, then your browser will reject the connection due to a failed hostname verification. Similarly, if a client makes an mTLS connection to ZooKeeper, and your client hostname is foo.example.org, then you must present a certificate with DN CN=foo.example.org (or a wildcard certificate using CN=*.example.org). If you don’t present a certificate with this DN, then ZooKeeper will reject the connection unless you also define a Subject Alternative Name (SAN) containing foo.example.org. In the absence of SASL authentication, you can have different brokers and CLI tools connect to ZooKeeper with mTLS from different hosts only if they all use the same DN, in which case they must also specify a SAN that matches their respective hostnames.

Important

If using mTLS only (without SASL) and specifying zookeeper.set.acl=true, do not use certificates with CN=hostname where hostname differs based on the location from which the request originates as a means to satisfy hostname verification. If you do, you may find that brokers cannot access ZooKeeper nodes. Note that the full DN is included in the ZooKeeper ACL, and ZooKeeper only authorizes what is in the ACL.

As an alternative to using the DN, you can specify the identity of mTLS clients by writing a class that extends org.apache.zookeeper.server.auth.X509AuthenticationProvider and overrides the method protected String getClientId(X509Certificate clientCert). Choose a scheme name and set authProvider.[scheme] in ZooKeeper to be the fully-qualified class name of the custom implementation. Then configure ssl.authProvider=[scheme] to use it.

To enable mTLS authentication for ZooKeeper you must configure both ZooKeeper and Kafka. The following example shows a ZooKeeper configuration that enables mTLS authentication:

secureClientPort=2182
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
ssl.keyStore.location=<path-to-zookeeper-keystore>
ssl.keyStore.password=<zookeeper-keystore-password>
ssl.trustStore.location=<path-to-zookeeper-truststore>
ssl.trustStore.password=<zookeeper-truststore-password>

Important

ZooKeeper does not support setting the key password in the ZooKeeper server keystore to a value different from the keystore password itself. Be sure to set the key password to be the same as the keystore password.

The following example shows a Kafka configuration that connects to ZooKeeper using mTLS authentication:

# Connect to the ZooKeeper port configured for TLS
zookeeper.connect=zk1:2182,zk2:2182,zk3:2182
# Required to use TLS-to-ZooKeeper (default is false)
zookeeper.ssl.client.enable=true
# Required to use TLS-to-ZooKeeper
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
# Define key/trust stores to use TLS-to-ZooKeeper; ignored unless zookeeper.ssl.client.enable=true
zookeeper.ssl.keystore.location=<path-to-kafka-keystore>
zookeeper.ssl.keystore.password=<kafka-keystore-password>
zookeeper.ssl.truststore.location=<path-to-kafka-truststore>
zookeeper.ssl.truststore.password=<kafka-truststore-password>
# Tells broker to create ACLs on znodes
zookeeper.set.acl=true

Important

ZooKeeper does not support setting the key password in the ZooKeeper client keystore (broker) to a value different from the keystore password itself. Be sure to set the key password to be the same as the keystore password. Also note that unlike ZooKeeper, Kafka does not use camel case names for TLS-related configurations (for example, Kafka uses zookeeper.ssl.keystore.location, while ZooKeeper uses ssl.keyStore.location).

Encrypting communication to ZooKeeper with TLS¶

ZooKeeper connections that use mTLS are encrypted. Beginning with ZooKeeper 3.5.7 (the version shipped with Kafka 2.5), ZooKeeper supports the server-side configuration ssl.clientAuth=none, which is case-insensitive; valid options are: want, need (the default), and none. When set to none, ZooKeeper allows clients to connect using a TLS-encrypted connection without presenting their own CA certificate.

The following (partial) Confluent Server broker configuration shows how to connect to ZooKeeper with TLS encryption only:

Note

ZooKeeper configurations in zookeeper.properties with explicit enumerated values–such as ssl.clientAuth–do not allow trailing whitespaces. If you include trailing spaces then you will get a value like " need ", (note the extra whitespaces), which is not among the set of valid values {want, need, none}.

# Connect to the ZooKeeper port configured for TLS
zookeeper.connect=zk1:2182,zk2:2182,zk3:2182
# Required to use TLS-to-ZooKeeper (default is false)
zookeeper.ssl.client.enable=true
# Required to use TLS-to-ZooKeeper
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
# Define trust store to use encryption-only TLS-to-ZooKeeper; ignored unless zookeeper.ssl.client.enable=true
# There is no need to set keystore information assuming ssl.clientAuth=none on ZooKeeper
zookeeper.ssl.truststore.location=<path-to-kafka-truststore>
zookeeper.ssl.truststore.password=<kafka-truststore-password>
# Tells broker to create ACLs on znodes (if using SASL authentication, otherwise do not set)
zookeeper.set.acl=true

Connecting to TLS-enabled ZooKeeper using CLI tools¶

You can connect to TLS-enabled ZooKeeper quorums using the CLI tools zookeeper-security-migration.sh, kafka-acls.sh, and kafka-configs.sh. When using one of these tools, place the tool’s TLS configuration in a file and refer to that file using --zk-tls-config-file <filename>. The file contains all of the ZooKeeper-related configuration options that a broker would use (except it uses a different keystore when using mTLS instead of TLS encryption only). Refer to mTLS authentication for an mTLS configuration example or Encrypting communication to ZooKeeper with TLS for an encryption-only example. The following tool invocation shows how to use --zk-tls-config-file <filename>:

kafka-acls.sh --zk-tls-config-file <filename>  --authorizer-properties zookeeper.connect=zk1:2182 --authorizer-properties zookeeper.set.acl=true --add --allow-principal User:Alice --operation Read --operation Write --topic test-topic

You can connect to TLS-enabled ZooKeeper quorums using the ZooKeeper shell as follows:

zookeeper-shell.sh zk1:2182 -zk-tls-config-file <filename>

Be sure to use a single dash (-) rather than double-dash (--) when identifying the file containing the TLS configuration for zookeeper-shell.sh.

ZooKeeper quorum mTLS authentication¶

You can enable mTLS authentication between the ZooKeeper servers. For details, refer to Quorum TLS.

Demo¶

For a working example of these security features in ZooKeeper, including mTLS and SASL with Digest-MD5, see the Confluent Platform demo.