Configure Authentication with Confluent for Kubernetes¶
Confluent Platform components are configured without authentication by default. This document presents the supported authentication concepts and describes how to configure authentication for Confluent Platform using Confluent for Kubernetes (CFK).
For more details on security concepts in Confluent Platform, see Security.
For a comprehensive tutorial scenario for configuring authentication, see Deploy Secure Confluent Platform.
Authentication overview¶
Authentication verifies the identity of users and applications connecting to Kafka and other Confluent components.
Authentication to access Kafka¶
CFK supports the following authentication mechanisms for client applications and Confluent Platform components to access Kafka:
- SASL/PLAIN authentication: Clients use a username/password for authentication. The username/passwords are stored server-side in Kubernetes Secrets.
- SASL/PLAIN with LDAP authentication: Clients use a username/password for authentication. The username/passwords are stored in an LDAP server.
- mTLS authentication: Clients use TLS certificates for authentication. The certificate has a CN that identifies the client application principal name.
Authentication to access ZooKeeper¶
CFK supports the following authentication mechanisms for Kafka to access ZooKeeper:
Authentication to access Confluent Platform components¶
CFK supports the following authentication mechanisms for the components, specifically Connect, ksqlDB, Schema Registry, and Confluent Control Center:
- Basic authentication: Username/password for authentication, where the username/passwords are stored in Kubernetes Secrets.
- mTLS authentication: Clients use TLS certificates for authentication.
- LDAP authentication (for Control Center only): User principals and password credentials are stored in an LDAP server.
Authentication to access MDS¶
CFK supports the following authentication mechanism for client applications and Confluent Platform components to access Metadata Service (MDS):
Configure authentication to access Kafka¶
SASL/PLAIN authentication¶
SASL/PLAIN is a simple username/password mechanism that is typically used with TLS network encryption to implement secure authentication.
The username is used as the authenticated principal, which can then be used in authorization.
Configure SASL/PLAIN as Kafka authentication mechanism¶
Configure a Kafka listener, in the Kafka custom resource (CR), to use SASL/PLAIN as the authentication mechanism:
spec:
...
listeners:
external:
authentication:
type: plain --- [1]
jaasConfig: --- [2]
secretRef: --- [3]
jaasConfigPassThrough: --- [4]
secretRef: --- [5]
directoryPathInContainer: --- [6]
[1] Required. Set to
plain
.One of [3], [5], or [6] is required.
[2] When you use
jaasConfig
, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates the JAAS config. This is the recommended way to configure SASL/PLAIN for Kafka.[3] Provide a Kubernetes Secret with a list of usernames and passwords.
The expected key (the file name) is
plain-users.json
.The value for the key (the data in the file) is:
{ "username1": "password1", "username2": "password2", "username3": "password3" }
[4] An alternate way to configure JAAS is to use
jaasConfigPassThrough
. If you have customizations, such as using a custom login handler, you can bypass the CFK automation and provide the configuration directly.[5] Provide a Kubernetes Secret with the expected key and the value.
The expected key (the file name) is
plain-jaas.conf
.The expected value for the key (the data in the file) is your JAAS config text. See this Confluent Platform doc for understanding JAAS configs. The following example uses the standard login module and specifies two additional users:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="admin-secret" \ user_admin="admin-secret" \ user_user1="user1-secret" \ user_user2=”user2-secret”;
[6] Provide the required credential from the directory path in the container.
The expected key (file name) is
plain-jaas.conf
. IfjaasConfigPassThrough.directoryPathInContainer
is configured as/vaults/secrets
, the expected file,plain-jaas.conf
, exists in the directory path.The following example for the expected value for the key (the data in the file) uses the standard login module and specifies two additional users:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="admin-secret" \ user_admin="admin-secret" \ user_user1="user1-secret" \ user_user2=”user2-secret”;
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.
Configure Confluent components to authenticate to Kafka using SASL/PLAIN¶
For each of the Confluent components that communicates with Kafka, configure SALS/PLAIN authentication in the component CR as below:
spec:
...
dependencies:
kafka:
authentication:
type: plain --- [1]
jaasConfig: --- [2]
secretRef: --- [3]
jaasConfigPassThrough: --- [4]
secretRef: --- [5]
directoryPathInContainer: --- [6]
[1] Required. Set to
plain
.One of [2] or [4] is required.
[2] When you use
jaasConfig
, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates JAAS config.[3] Provide a Kubernetes Secret with the username/password used to authenticate to Kafka.
The expected key (filename) is
plain.txt
.The expected value for the key (the data in the file) is:
username=<username> password=<password>
[4] An alternate way to configure JAAS is to use
jaasConfigPassThrough
. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly.[5] Provide a Kubernetes Secret with the expected key and the value.
The expected key (filename) is
plain-jaas.conf
.The expected value for the key (the data in the file) is your JAAS config text. See this Confluent Platform doc for understanding JAAS configs. The following example uses the standard login module and specifies two additional users:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="admin-secret" \ user_admin="admin-secret" \ user_user1="user1-secret" \ user_user2="user2-secret";
[6] Provide the required credential from the directory path in the container.
The expected key (file name) is
plain-jaas.conf
.If
jaasConfigPassThrough.directoryPathInContainer
is configured as/vaults/secrets
, the expected file,plain-jaas.conf
, exists in the directory path. The following example for the expected value for the key (the data in the file) uses the standard login module and specifies two additional users:sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="admin-secret" \ user_admin="admin-secret" \ user_user1="user1-secret" \ user_user2="user2-secret";
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.
SASL/PLAIN with LDAP authentication¶
SASL/PLAIN with LDAP callback handler is a variation of SASL/PLAIN. When you use SASL/PLAIN with LDAP for authentication, the username principals and passwords are retrieved from an LDAP server.
Configure SASL/PLAIN with LDAP as Kafka authentication mechanism¶
You must set up an LDAP server, for example, AD, before configuring and starting up a Kafka cluster with the SASL/PLAIN with LDAP authentication. For more information, see Configuring Kafka Client Authentication with LDAP.
Note
If you configure any Kafka listeners with the SASL/PLAIN authentication mode
along with a SASL/PLAIN LDAP listener, you must use
authentication.jaasConfigPassThrough
for the SASL/PLAIN listener with
PlainLoginModule
. You cannot use authentication.jaasConfig
for the
SASL/PLAIN listener, and you cannot use FileBasedLoginModule
.
To use SASL/PLAIN with LDAP as the authentication mechanism:
Configure the listeners in the Kafka custom resource (CR):
spec: listeners: internal: authentication: type: ldap --- [1] jaasConfig: --- [2] secretRef: --- [3] jaasConfigPassThrough: --- [4] secretRef: --- [5] directoryPathInContainer:--- [6] external: authentication: type: ldap --- [7] custom: authentication: type: ldap --- [8]
[1] Required for the SASL/PLAIN with LDAP authentication for the internal Kafka listeners.
[2] When you use
jaasConfig
to pass credentials, you provide the user name and password, and CFK automates configuration. When you add, remove, or update the user, CFK automatically updates the JAAS configuration. This is the recommended way to configure SASL/PLAIN LDAP for Kafka.[3] For
jaasConfig
, provide a Kubernetes Secret with the user name and password for inter-broker authentication.The expected key (the file name) is
plain-interbroker.txt
.The value for the key (the data in the file) is:
username=kafka password=kafka-secret
The username and password must belong to a user that exists in LDAP. This is the user that each Kafka broker authenticates when the cluster starts.
[4] An alternate way to configure JAAS is to use
jaasConfigPassThrough
. If you have customizations, such as using a custom login handler, you can bypass the CFK automation and provide the configuration directly.[5] Provide a Kubernetes Secret with the expected key and the value.
The expected key (the file name) is
plain-jaas.conf
.The expected value for the key (the data in the file) is your JAAS config text:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="kafka" \ password="kafka-secret";
See this Confluent Platform document for understanding JAAS configs.
[6] Provide the required credential from the directory path in the container.
The expected key (file name) is
plain-jaas.conf
. IfjaasConfigPassThrough.directoryPathInContainer
is configured as/vaults/secrets
, the expected file,plain-jaas.conf
, exists in the directory path.The expected value for the key (the data in the file) is your JAAS config text:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="kafka" \ password="kafka-secret";
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
[7] Required for the SASL/PLAIN with LDAP authentication for the external Kafka listeners.
[8] Required for the SASL/PLAIN with LDAP authentication for the custom Kafka listeners.
[7] [8] To configure authentication type
ldap
on external or custom listeners, you do not need to specifyjaasConfig
orjaasConfigPassThrough
.
Configure the identity provider in the Kafka CR:
spec: identityProvider: --- [1] type: ldap --- [2] ldap: --- [3] address: --- [4] authentication: --- [5] type: --- [6] simple: --- [7] tls: enabled: --- [8] configurations: --- [9]
[1] Required for the Kafka authentication type
ldap
. Specifies the identity provider configuration.When the MDS is enabled, this property is ignored, and the LDAP configuration in
spec.services.mds.provider
is used.[2] Required.
[3] This block includes the same properties used in the
spec.services.mds.provider.ldap
block in this Kafka CR.[4] Required. The address of the LDAP server, for example,
ldaps://ldap.confluent.svc.cluster.local:636
.[5] Required. The authentication method to access the LDAP server.
[6] Required. Specify
simple
ormtls
.[7] Required if the authentication type ([6]) is set to
simple
.[8] Required if the authentication type ([6]) is set to
mtls
. Set totrue
.[9] Required. The LDAP configuration settings.
Apply the configuration:
kubectl apply -f <Kafka CR>
Configure Confluent components to authenticate to Kafka using SASL/PLAIN with LDAP¶
When Kafka is configured with SASL/PLAIN with LDAP, Confluent components and clients authenticate to Kafka as SASL/PLAIN clients. The clients must authenticate as users in LDAP.
See Configure Confluent components to authenticate to Kafka using SASL/PLAIN for configuration details.
mTLS authentication¶
Configure mTLS as Kafka authentication mechanism¶
mTLS utilizes TLS certificates as an authentication mechanism. The certificate provides the identity.
The certificate Common Name (CN) is used as the authenticated principal, which can then be used in authorization.
Configure a Kafka listener as below, in the Kafka CR, to use mTLS as the authentication mechanism:
spec:
...
listeners:
external:
authentication:
type: mtls --- [1]
principalMappingRules:
- RULE:.*CN[\s]?=[\s]?([a-zA-Z0-9.]*)?.*/$1/ --- [2]
tls:
enabled: true --- [3]
[1] Required. Set to
mTLS
.[2] Optional. This specifies a mapping rule that extracts the principal name from the certificate Common Name (CN).
The regular expression (regex) used in the mapping rule is Java mapping API. You can use a site, such as Regular Expression Test Drive, to validate the regex.
[3] Required for mTLS authentication. Set to
true
.
Configure Confluent components to authenticate to Kafka using mTLS¶
For each of the Confluent components that communicates with Kafka, configure the mTLS authentication mechanism in the component CR as below:
spec:
...
dependencies:
kafka:
authentication:
type: mtls --- [1]
tls:
enabled: true --- [2]
- [1] Required. Set to
mtls
. - [2] Required for mTLS authentication. Set to
true
.
Configure authentication to access ZooKeeper¶
SASL/DIGEST authentication¶
Configure DIGEST as ZooKeeper authentication¶
ZooKeeper supports authentication using the SASL DIGEST-MD5 mechanism.
Enable DIGEST authentication in the ZooKeeper CR as below:
spec:
...
authentication:
type: digest --- [1]
jaasConfig: --- [2]
secretRef: --- [3]
jaasConfigPassThrough: --- [4]
secretRef: --- [5]
directoryPathInContainer: --- [6]
[1] Required. Set to
digest
.One of [2], [3], or [4] is required.
[2] When you use
jaasConfig
, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates JAAS config. This is the recommended way to configure SASL/DIGEST for ZooKeeper.[3] Provide a Kubernetes Secret with the username/passwords.
The expected key (the file name) is
digest-users.json
.The expected value for the key (the data in the file) is:
{ "username1": "password1", "username2": "password2" }
[4] An alternate way to configure JAAS is to use
jaasConfigPassThrough
. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly.[5] Provide a Kubernetes Secret with the expected key and the value. The expected key (the file name) is
digest-jaas.conf
. The expected value for the key (data in the file) is:Server { org.apache.zookeeper.server.auth.DigestLoginModule required user_super="adminsecret" user_user1="user1-secret"; };
[6] Provide the required credential from the directory path in the container. The expected key/file name is
digest-jaas.conf
.If
jaasConfigPassThrough.directoryPathInContainer
is configured as/vaults/secrets
, the expected file,digest-jaas.conf
, exists in the directory path. The following example for the expected value for the key (the data in the file) uses the standard login module and specifies one additional user:Server { org.apache.zookeeper.server.auth.DigestLoginModule required user_super="adminsecret" user_bob="bob-secret"; };
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.
Configure Kafka to authenticate to ZooKeeper using digest¶
For Kafka to authenticate to ZooKeeper using digest authentication, configure the Kafka CR as below:
spec:
...
dependencies:
zookeeper:
authentication:
type: digest --- [1]
jaasConfig: --- [2]
secretRef: --- [3]
jaasConfigPassThrough: --- [4]
secretRef: --- [5]
directoryPathInContainer: --- [6]
[1] Required. Set to
digest
.One of [2], [3], or [4] is required.
[2] When you use
jaasConfig
, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates the JAAS config. This is the recommended way to configure SASL/DIGEST for ZooKeeper.[3] Provide a Kubernetes Secret with the username/password used to authenticate to ZooKeeper.
The expected key (the file name) is
digest.txt
.The value for the key (the data in the file) is shown below. Double quotes are required around the username and password.
username="<username>" password="<password>"
[4] An alternate way to configure JAAS is to use
jaasConfigPassThrough
. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly.[5] Provide a Kubernetes Secret with the expected key and the value.
The expected key and filename is
digest-jaas.conf
.The following is an example value (the data in the file) for
digest-jaas.conf
with a standard login module and a user. The expected value for the key (the data in the file) is:Client { //zookeeper dependencies org.apache.zookeeper.server.auth.DigestLoginModule required username="bob" password="password"; };
[6] Provide the path where required credentials are injected by Vault. See [5] above for the expected key and the value.
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.
mTLS authentication¶
Configure mTLS as ZooKeeper authentication¶
Enable mTLS authentication in the ZooKeeper CR as below:
spec:
...
authentication:
type: mtls
Configure Kafka to authenticate to ZooKeeper using mTLS¶
For Kafka to authenticate to ZooKeeper using mTLS authentication, confgure the Kafka CR as below:
spec:
...
dependencies:
zookeeper:
authentication:
type: mtls --- [1]
tls:
enabled: true --- [2]
- [1] Required. Set to
mtls
. - [2] Required for mTLS authentication. Set to
true
.
Configure authentication to access Confluent Platform components¶
Basic authentication¶
Configure basic authentication for Confluent Platform components¶
The credentials are stored on the server side, in the following format:
<username>: <password>, <role that user is assigned to>
The following roles are supported, by default:
- For REST Proxy: The
admin
,developer
,user
, andkrp-user
roles are available. - For ksqlDB: The
admin
,developer
,user
, andksql-user
roles are available. - For Schema Registry: The
admin
,developer
,user
, andsr-user
roles are available. - For Confluent Control Center: The
Administrators
andRestricted
roles are available.
Warning
For Connect, there is no support for roles.
Configure basic authentication in the component CR as below:
spec:
...
authentication:
type: basic --- [1]
basic:
secretRef: --- [2]
directoryPathInContainer: --- [3]
restrictedRoles: --- [4]
roles: --- [5]
[1] Required. Set to
basic
.[2] or [3] Required.
[2] Provide a Kubernetes Secret with the username/passwords.
The expected key is
basic.txt
.The value for the key is:
username1: password1,role1 username2: password2,role2
[3] Provide the path where required credentials are injected by Vault. See [2] above for the expected key and the value.
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.[4] Optional. A list of restricted roles on the server in Confluent Control Center.
[5] Optional. A list of roles on the server side.
Configure Confluent components to authenticate to other Confluent Platform component using basic authentication¶
The username/password for basic authentication are either loaded through the
secretRef
or through directoryPathInContainer
.
Warning
Confluent Control Center does not support connecting to Connect and ksqlDB using basic authentication.
Enable basic authentication in the component CR as below. <component>
is
the Confluent Platform component that this component needs to authenticate to:
spec:
...
dependencies:
<component>:
url:
authentication:
type: basic --- [1]
basic:
secretRef: --- [2]
directoryPathInContainer --- [3]
[1] Required. Set to
basic
.[2] or [3] are required.
[2] The username and password are loaded through
secretRef
.The expected key is is
basic.txt
.The value for the key is:
username=<username> password=<password>
[3] The username and password are loaded through Vault. See [2] above for the expected key and the value.
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
mTLS authentication¶
Configure mTLS authentication for Confluent Platform components¶
Configure mTLS authentication in the component CR as below:
spec:
...
authentication:
type: mtls --- [1]
- [1] Required. Set to
mtls
.
Configure Confluent components to authenticate to other Confluent Platform components using mTLS authentication¶
Enable mTLS authentication in the component CR as below. <component>
is the
Confluent Platform component that this component needs to authenticate to:
spec:
...
dependencies:
<component>:
url:
authentication:
type: mtls --- [1]
tls:
enabled: true --- [2]
- [1] Required. Set to
mtls
. - [2] Required for mTLS authentication. Set to
true
.
LDAP authentication for Confluent Control Center¶
In addition to basic and mTLS authentication methods, Confluent Control Center supports LDAP as an authentication method.
You can configure the Control Center CR to pull users and groups from an LDAP server:
kind: ControlCenter
spec:
authentication:
type: ldap --- [1]
ldap:
secretRef: --- [2]
roles: --- [3]
restrictedRoles: --- [4]
property: --- [5]
# useLdaps: "true"
# contextFactory: "com.sun.jndi.ldap.LdapCtxFactory"
# hostname: ""
# port: "389"
# bindDn: ""
# bindPassword: ""
# authenticationMethod: ""
# forceBindingLogin: "true"
# userBaseDn: ""
# userRdnAttribute: "sAMAccountName"
# userIdAttribute: "sAMAccountName"
# userPasswordAttribute: "userPassword"
# userObjectClass: "user"
# roleBaseDn: ""
# roleNameAttribute: "cn"
# roleMemberAttribute: "member"
# roleObjectClass: "group"
[1] Required. Set to
ldap
.[2] Secret for LDAP credentials.
The expected key is
ldap.txt
.The value for the key is:
username=<bindDn_value> password=<bindPassword_value>
If you have security considerations, pass empty values for
bindDn
andbindPassword
in the CR. The CFK will replace them and add them as appropriately.For the password for
bindDn
, escape any restricted LDAP characters. For best results, avoid characters that require escaping. Follow Best Practices for LDAP Naming Attributes.[3] Optional. By default, it’s set to
["Administrators", "Restricted"]
.[4] Optional. List of roles with limited read-only access. No editing or creating allowed in Control Center.
[5] Required. See Configure LdapLoginModule for details.
LDAP over SSL authentication for Confluent Control Center¶
When configuring Control Center for encryption and authentication using LDAP over
SSL (LDAPS), the LDAPS truststore must be exported to the JVM using the
configOverrides
feature.
This requirement does not apply to RBAC-enabled environments where Control Center relies on the external MDS for authentication.
Add the LDAP SSL certificate to the JVM truststore in the Confluent Control Center CR as shown below:
spec:
configOverrides:
jvm:
-Djavax.net.ssl.trustStore=<path to truststore.jks>
-Djavax.net.ssl.trustStorePassword=<password for the truststore>
Configure authentication to access MDS¶
Bearer authentication¶
For each of the Confluent components that communicate with MDS, configure bearer authentication mechanism in the component CR:
spec:
dependencies:
mds:
authentication:
type: bearer --- [1]
bearer:
secretRef: --- [2]
[1] Required. Set to
bearer
.[2] Required.
The username and password are loaded through
secretRef
.The expected key is
bearer.txt
.The value for the key is:
username=<username> password=<password>
When RBAC is enabled (spec.authorization.type: rbac
), CFK always uses the
Bearer authentication for Confluent components, ignoring the
spec.authentication
setting. It is not possible to set the component
authentication type to mTLS when RBAC is enabled.