Configure Authentication for Confluent Platform Components Using Confluent for Kubernetes¶
Confluent Platform components are configured without authentication by default. This document presents the supported authentication concepts and describes how to configure authentication for Confluent Platform using Confluent for Kubernetes (CFK).
For more details on security concepts in Confluent Platform, see Security in Confluent Platform.
For a comprehensive tutorial scenario for configuring authentication, see Deploy Secure Confluent Platform.
Configure authentication to access ZooKeeper¶
This section describes the following authentication mechanisms for Kafka to access ZooKeeper:
SASL/DIGEST authentication¶
Server-side SASL/DIGEST authentication for ZooKeeper¶
ZooKeeper supports authentication using the SASL DIGEST-MD5 mechanism.
You can use the JAAS and JAAS pass-through mechanisms to set up the credentials.
Create server-side SASL/DIGEST credentials using JAAS config¶
When you use jaasConfig
, you provide the user names and passwords, and CFK
automates configuration. For example, when you add, remove, or update users,
CFK automatically updates JAAS config. This is the recommended way to configure
SASL/DIGEST for ZooKeeper.
The expected key for the server-side SASL/DIGEST credential is
digest-users.json
.
Create a
.json
file and add the expected value, in the following format:{ "username1": "password1", "username2": "password2" }
Create a Kubernetes secret using the expected key (
digest-users.json
) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./digest-users.json
file that contains the credentials:kubectl create secret generic credential \ --from-file=digest-users.json=./digest-users.json \ --namespace confluent
Create server-side SASL/DIGEST credentials using JAAS config pass-through¶
An alternate way to configure JAAS is to use jaasConfigPassThrough
. If you
have customizations, such as using custom login handlers, you can bypass the
CFK automation and provide the configuration directly using
jaasConfigPassThrough
.
The expected server-side key for jaasConfigPassThrough
is
digest-jaas.conf
.
Create a
.conf
file and add the expected value. For example:Server { org.apache.zookeeper.server.auth.DigestLoginModule required user_super="adminsecret" user_user1="user1-secret"; };
You can use a Kubernetes secret or a directory path in the container to store the credentials.
Create a Kubernetes Secret with the expected key (
digest-jaas.conf
) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./digest-users.conf
file that contains the credentials:kubectl create secret generic credential \ --from-file=digest-jaas.conf=./digest-users.conf \ --namespace confluent
Use a directory path in the container to provide the required credentials.
If
jaasConfigPassThrough.directoryPathInContainer
is configured as/vaults/secrets
in the ZooKeeper CR, the expected file,digest-jaas.conf
, must exist in that directory path.See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.
Configure ZooKeeper for server-side SASL/DIGEST authentication¶
Enable the server-side SASL/DIGEST authentication in the ZooKeeper CR as below:
kind: Zookeeper
spec:
authentication:
type: digest --- [1]
jaasConfig: --- [2]
secretRef: --- [3]
jaasConfigPassThrough: --- [4]
secretRef: --- [5]
directoryPathInContainer: --- [6]
- [1] Required. Set to
digest
. - [2] When you use
jaasConfig
, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates JAAS config. This is the recommended way to configure SASL/DIGEST for ZooKeeper. - One of [3], [5], or [6] is required. Only specify one.
- [3] Provide the name of the Kubernetes secret that you created in the previous section.
- [4] An alternate way to configure JAAS is to use
jaasConfigPassThrough
. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly. - [5] Provide a Kubernetes secret that you created in the previous section.
- [6] Provide the directory path in the container you set up in the
previous section. The expected
file,
digest-jaas.conf
, must exist in the directory path.
Client-side SASL/DIGEST authentication for ZooKeeper¶
Configure client-side Kafka to authenticate to ZooKeeper using SASL/DIGEST.
You can use the JAAS and JAAS pass-through mechanisms to set up the credentials.
Create client-side SASL/DIGEST credentials using JAAS config¶
When you use jaasConfig
, you provide the user names and passwords, and CFK
automates configuration. For example, when you add, remove, or update users,
CFK automatically updates the JAAS config. This is the recommended way to
configure SASL/DIGEST for ZooKeeper.
The expected client-side key for jaasConfig
is digest.txt
.
Create a
.txt
file and add the expected value, in the following format:username=<user> password=<password>
Create a Kubernetes Secret with the expected key (
digest.txt
) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./digest-users.txt
file that contains the credentials:kubectl create secret generic credential \ --from-file=digest.txt=./digest-users.txt \ --namespace confluent
Create client-side SASL/DIGEST credentials using JAAS config pass-through¶
If you have customizations, such as using custom login handlers, you can bypass
the CFK automation and provide the configuration directly using
jaasConfigPassThrough
.
The expected server-side key for jaasConfigPassThrough
is
digest-jaas.conf
.
Create a
.conf
file and add the expected value to the file.The following is an example value (the data in the file) for
digest-jaas.conf
with a standard login module and a user,bob
.Client { //zookeeper dependencies org.apache.zookeeper.server.auth.DigestLoginModule required username="bob" password="password"; };
You can use a Kubernetes secret or a directory path in the container to store the credentials.
Create a Kubernetes secret using the expected key (
digest-jaas.conf
) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./digest-jaas-users.conf
file that contains the credentials:kubectl create secret generic credential \ --from-file=digest-jaas.conf=./digest-jaas-users.conf \ --namespace confluent
Use a directory path in the container to provide the required credentials.
If
jaasConfigPassThrough.directoryPathInContainer
is configured as/vaults/secrets
in the Kafka CR, the expected file,digest-jaas.conf
, must exist in that directory path.See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.
Configure Kafka for client-side SASL/DIGEST authentication¶
For Kafka to authenticate to ZooKeeper using SASL/DIGEST authentication, configure the Kafka CR as below:
kind: Kafka
spec:
dependencies:
zookeeper:
authentication:
type: digest --- [1]
jaasConfig: --- [2]
secretRef: --- [3]
jaasConfigPassThrough: --- [4]
secretRef: --- [5]
directoryPathInContainer: --- [6]
[1] Required. Set to
digest
.[2] When you use
jaasConfig
, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates the JAAS config. This is the recommended way to configure SASL/DIGEST for ZooKeeper.One of [3], [5], or [6] is required. Only specify one.
[3] Provide the name of the Kubernetes secret you set up in the previous section for Kafka to authenticate to ZooKeeper.
[4] An alternate way to configure JAAS is to use
jaasConfigPassThrough
. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly.[5] Provide the name of the Kubernetes secret you set up in the previous section for Kafka to authenticate to ZooKeeper.
[6] Provide the container directory path you set up in the previous section.
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.
mTLS authentication¶
Server-side mTLS authentication for ZooKeeper¶
Enable mTLS authentication in the ZooKeeper CR as below:
kind: Zookeeper
spec:
authentication:
type: mtls
Client-side mTLS authentication ZooKeeper¶
For Kafka to authenticate to ZooKeeper using mTLS authentication, configure the Kafka CR as below:
kind: Kafka
spec:
dependencies:
zookeeper:
authentication:
type: mtls --- [1]
tls:
enabled: true --- [2]
- [1] Required. Set to
mtls
. - [2] Required for mTLS authentication. Set to
true
.
Configure authentication to access other Confluent Platform components¶
This section describes the following authentication methods for the Confluent Platform components (other than Kafka and ZooKeeper):
- OAuth/OIDC authentication
- Basic authentication
- mTLS authentication
- LDAP authentication (for Control Center only)
- Single sign-on (for Control Center only)
Basic authentication¶
Server-side basic authentication for Confluent components¶
Create server-side basic credentials¶
The expected server-side key for basic authentication is basic.txt
.
Create a
.txt
file and add the expected value, in the following format:<username1>: <password1>, <role that username1 is assigned to> ... <usernameN>: <passwordN>, <role that usernameN is assigned to>
To be authorized to access a Confluent Platform component, an authenticated user must belong to one of the basic authentication roles. The Roles associated with a user does not differentiate permissions, except Confluent Control Center where the users belonging to restricted roles have limited permissions as stated in Control Center Security.
The following default basic authentication roles are supported:
- For REST Proxy: The
admin
,developer
,user
, andkrp-user
roles are available. - For ksqlDB: The
admin
,developer
,user
, andksql-user
roles are available. - For Schema Registry: The
admin
,developer
,user
, andsr-user
roles are available. - For Confluent Control Center: The
Administrators
andRestricted
roles are available.
Warning
For Connect, there is no support for roles.
- For REST Proxy: The
You can use a Kubernetes secret or a directory path in the container to store the basic credentials.
Create a Kubernetes secret using the expected key (
basic.txt
) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./creds-basic.txt
file that contains the credentials:kubectl create secret generic credential \ --from-file=basic.txt=./creds-basic.txt \ --namespace confluent
Use a directory path in the container to provide the required credentials.
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
See CFK GitHub examples for more information on using the
directoryPathInContainer
property with Vault.
Configure Confluent component for server-side basic authentication¶
Configure the server-side basic authentication in the component CR as below:
kind: <Confluent component>
spec:
authentication:
type: basic --- [1]
basic:
secretRef: --- [2]
directoryPathInContainer: --- [3]
restrictedRoles: --- [4]
roles: --- [5]
[1] Required. Set to
basic
.[2] or [3] Required. Do not specify both.
[2] Provide the name of the Kubernetes secret you created in the previous section.
[3] Provide the path in the container in the previous section. The expected file,
basic.txt
, must exist in the specified directory path.[4] Optional. A list of restricted roles on the server in Confluent Control Center. The setting only applies to Control Center and is ignored for other Confluent Platform components.
For Control Center, the users who belong to restricted roles have limited permissions as stated in Control Center Security.
[5] Optional. A list of roles on the server side. To be authorized to access this Confluent Platform component, users must belong to one of these roles. See Create server-side basic credentials for the component default roles.
Client-side basic authentication for Confluent components¶
Configure client-side Confluent components to authenticate to other Confluent Platform component using basic authentication.
Create client-side basic credentials¶
The expected client-side key for basic authentication is basic.txt
.
Create a
.txt
file and add the expected value, in the following format:username=<username> password=<password>
The username/password for basic authentication are either loaded through the
secretRef
or throughdirectoryPathInContainer
.
Configure Confluent components for client-side basic authentication¶
Enable the client-side basic authentication in the component CR as below.
<component>
is the Confluent Platform component that this component needs to authenticate
to:
kind: <this Confluent component>
spec:
dependencies:
<component>:
url:
authentication:
type: basic --- [1]
basic:
secretRef: --- [2]
directoryPathInContainer --- [3]
[1] Required. Set to
basic
.[2] or [3] is required. Do not specify both.
[2] Provide the name of the Kubernetes secret that you created in the previous section.
[3] Provide the directory in the container in the previous section.
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
mTLS authentication¶
Server-side mTLS authentication for Confluent components¶
Configure mTLS authentication in the Confluent component CR as below:
kind: <Confluent component>
spec:
authentication:
type: mtls --- [1]
- [1] Required. Set to
mtls
.
Client-side mTLS authentication for Confluent components¶
To configure Confluent components to authenticate to other Confluent Platform components using
mTLS authentication, enable mTLS authentication in the component CR as below.
<component>
is the Confluent Platform component that this component needs to authenticate
to:
kind: <this client Confluent component>
spec:
dependencies:
<component>:
url:
authentication:
type: mtls --- [1]
tls:
enabled: true --- [2]
- [1] Required. Set to
mtls
. - [2] Required for mTLS authentication. Set to
true
.
OAuth/OIDC authentication¶
Open Authentication (OAuth) is an open-standard authorization protocol that provides applications the ability for securely designated access. You can leverage your own identity provider and centralize identity management across your Confluent Platform and other service deployments on the cloud and on-premises.
Starting with CFK 2.9 and Confluent Platform 7.7, you can configure Confluent components with OAuth/OIDC, an OAuth-based authentication mechanism.
OAuth is not supported for ksqlDB.
Server-side OAuth/OIDC authentication for Confluent components¶
Configure the server-side OAuth/OIDC authentication in the component CR as following. Server-side OAuth/OIDC is supported for Connect, REST Proxy, and Schema Registry.
kind: <this client Confluent component>
spec:
authentication:
type: oauth --- [1]
oauth:
secretRef: --- [2a]
directoryPathInContainer: --- [2b]
configuration:
groupsClaimName: --- [3]
subClaimName: --- [4]
audience: --- [5]
expectedIssuer: --- [6]
jwksEndpointUri: --- [7]
tokenEndpointUri: --- [8]
scope: --- [9]
loginConnectTimeoutMs: --- [10]
loginReadTimeoutMs: --- [11]
loginRetryBackoffMs: --- [12]
loginRetryMaxBackoffMs: --- [13]
[1] Required. Set to
oauth
.[2a] or [2b] Specify only one setting.
[2a] The secret that contains the OIDC client ID and the client secret for authorization and token request to the identity provider.
Create the secret that contains two keys with their respective values,
clientId
andclientSecret
as following:clientId=<client-id> clientSecret=<client-secret>
[2b] The OIDC client ID and the client secret for authorization and token request to the identity provider (IDP).
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
[3] Required. The name of the claim in token for identifying the groups of subject in the JSON Web Tokens (JWT). The default value is
groups
.[4] The subject name of the JWT (session token). The default value is
sub
. Used in SSO.[5] Required. The audience claim in the JWT payload.
[6] The issuer URL, which is typically the authorization server’s URL. This value is used to compare to the issuer claim in the JWT for verification.
[7] The JSON Web Key Set (JWKS) URI. It is used to verify any JWT issued by the IDP.
[8] The base endpoint URI for the authorize that initiates an OAuth authorization request.
[9] Required only when your identity provider does not have a default scope or your groups claim is linked to a scope.
[10] Connect timeout with IDP in ms.
[11] Read timeout with IDP in ms.
[12] Retry backoff with IDP in ms.
[13] Max retry backoff with IDP in ms
Client-side OAuth/OIDC authentication for Confluent components¶
Configure client-side Confluent component (client-component
) to authenticate
to other Confluent Platform component (server-component
) using OAuth/OIDC authentication.
kind: <client-component>
spec:
dependencies:
<server-component>:
authentication:
type: oauth --- [1]
oauth:
secretRef: --- [2a]
directoryPathInContainer: --- [2b]
configuration:
tokenEndpointUri: --- [3]
[1] Required. Set to
oauth
.[2a] or [2b] Specify only one setting.
[2a] The secret that contains the OIDC client ID and the client secret for authorization and token request to the identity provider.
Create the secret that contains two keys with their respective values,
clientId
andclientSecret
as following:clientId=<client-id> clientSecret=<client-secret>
[2b] The OIDC client ID and the client secret for authorization and token request to the IDP.
See Provide secrets for Confluent Platform component CR for providing the credential and required annotations when using Vault.
[3] Required for OAuth for inter broker communication. The base endpoint URI for the authorize that initiates an OAuth authorization request.
For the full list of the OAuth settings, see OAuth configuration.
LDAP authentication for Confluent Control Center¶
Confluent Control Center supports LDAP as an authentication method.
Create LDAP credentials¶
The expected server-side key for LDAP authentication is ldap.txt
.
Create a
.txt
file and add the expected value, in the following format:username=<bindDn_value> password=<bindPassword_value>
For the password for
bindDn
, escape any restricted LDAP characters. For best results, avoid characters that require escaping.Create a Kubernetes secret using the expected key (
ldap.txt
) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./creds-ldap.txt
file that contains the credentials:kubectl create secret generic credential \ --from-file=ldap.txt=./creds-ldap.txt \ --namespace confluent
Configure Confluent Control Center for server-side LDAP authentication¶
Configure the Control Center CR to pull users and groups from an LDAP server:
kind: ControlCenter
spec:
authentication:
ldap:
secretRef: --- [1]
roles: --- [2]
restrictedRoles: --- [3]
property: --- [4]
# useLdaps: "true"
# contextFactory: "com.sun.jndi.ldap.LdapCtxFactory"
# hostname: ""
# port: "389"
# bindDn: "" --- [5]
# bindPassword: "" --- [6]
# authenticationMethod: ""
# forceBindingLogin: "true"
# userBaseDn: ""
# userRdnAttribute: "sAMAccountName"
# userIdAttribute: "sAMAccountName"
# userPasswordAttribute: "userPassword"
# userObjectClass: "user"
# roleBaseDn: ""
# roleNameAttribute: "cn"
# roleMemberAttribute: "member"
# roleObjectClass: "group"
- [1] Provide the Kubernetes secret you created in the previous section for LDAP credentials.
- [2] Optional. A list of roles on the server side. To be authorized to access
Confluent Control Center, users must belong to one of these roles. By default, it’s set to
["Administrators", "Restricted"]
. - [3] Optional. List of roles with limited read-only access. No editing or creating allowed in Control Center.
- [4] Required. See Configure LdapLoginModule for details.
- [5] [6] If you have security considerations, pass empty values for
bindDn
andbindPassword
in the CR. The CFK will replace them and add them as appropriately.
LDAP over SSL authentication for Confluent Control Center¶
When configuring Control Center for encryption and authentication using LDAP over
SSL (LDAPS), the LDAPS trust store must be exported to the JVM using the
configOverrides
feature.
This requirement does not apply to RBAC-enabled environments where Control Center relies on the external MDS for authentication.
Add the LDAP SSL certificate to the JVM trust store in the Confluent Control Center CR as shown below:
kind:ControlCenter
spec:
configOverrides:
jvm:
-Djavax.net.ssl.trustStore=<path to truststore.jks>
-Djavax.net.ssl.trustStorePassword=<password for the truststore>
Single sign-on authentication for Confluent Control Center¶
In CFK, you can configure single sign-on (SSO) authentication for Control Center using OpenID Connect (OIDC). OIDC is an identity layer that allows third-party applications to verify the identity of the end user.
To enable SSO to Control Center:
In the Kafka CR, configure MDS and the MDS identity provider, LDAP or SSO.
kind: Kafka spec: services: mds: --- [1] tokenKeyPair: --- [2] secretRef: directoryPathInContainer: provider: ldap: --- [3a] address: authentication: configurations: oidc: --- [4] sso: --- [3b] clientCredentials: secretRef: --- [5] configurations: groupsClaimName: --- [6] subClaimName: --- [7] groupsClaimScope: --- [8] issuer: --- [9] jwksEndpointUri: --- [10] authorizeBaseEndpointUri: --- [11] tokenBaseEndpointUri: --- [12] refreshToken: --- [13] sessionTokenExpiry: --- [14] sessionMaxTimeout: --- [15]
[1] Required.
[2] Required. The token key pair to authenticate to the MDS. Use
secretRef
ordirectoryPathInContainer
to specify.You need to add the public key and the token key pair to the secret or directoryPathInContainer. For details, see Create a PEM key pair for MDS.
An example command to create a secret is:
kubectl create secret generic mds-token \ --from-file=mdsPublicKey.pem=mds-publickey.txt \ --from-file=mdsTokenKeyPair.pem=mds-tokenkeypair.txt \ --namespace confluent
[3a] or [3b] is required. See the snippet below for example configurations.
[4] Deprecated.
For backward compatibility, you can use the setting to configure the same set of properties as
sso
. However, thesso
setting is recommended for SSO service configuration.[5] The secret that contains a OIDC client ID and the client secret for authorization and token request to IDP.
Create the secret with the file name
oidcClientSecret.txt
that contains two keys with their respective values,clientId
andclientSecret
.[6] Groups in JSON Web Tokens (JWT). The default value is
groups
.JWT (Session Token) is issued by Confluent (MDS) to maintain the session. This contains information passed by IDP in id token along with custom claims (if any) added by the MDS.
[7] The subject name of the JWT (session token). The default value is
sub
.[8] If any additional scope is needed to include groups in the JWT, set the optional variable based on the IDP. The setting lets the MDS pass these scope(s) during the authorization process.
Possible values are
groups
,openid
,offline_access
, etc.[9] The issuer URL, which is typically the authorization server’s URL. This value is used to compare to the issuer claim in the JWT for verification.
[10] The JSON Web Key Set (JWKS) URI. It is used to verify any JWT issued by the IDP.
[11] The base endpoint URI for the authorize that initiates an OAuth authorization request.
[12] The IDP token endpoint from where a token is requested by the MDS.
[13] Specify whether offline access scope would be requested in the authorization URI. Set this to
false
if offline tokens are not allowed for the user or client in the IDP.[14] Time before the JWT (session token) needs to be renewed. Default value 15 minutes.
The actual expiry would be the lesser value between the ID token lifetime ([15]) value and the value of this setting.
[15] Max time up to which JWT (session token) could be renewed.
An example snippet of a Kafka CR configured with MDS and LDAP as the MDS identity provider:
kind: Kafka spec: services: mds: tls: enabled: true tokenKeyPair: secretRef: mds-token provider: ldap: address: ldaps://ldap.operator.svc.cluster.local:636 authentication: type: simple simple: secretRef: credential tls: enabled: true configurations: groupNameAttribute: cn groupObjectClass: group groupMemberAttribute: member groupMemberAttributePattern: CN=(.*),DC=test,DC=com groupSearchBase: dc=test,dc=com userNameAttribute: cn userMemberOfAttributePattern: CN=(.*),DC=test,DC=com userObjectClass: organizationalRole userSearchBase: dc=test,dc=com sso: clientCredentials: secretRef: oidc-client // to pass oidc client id and client secret configurations: groupsClaimName: groups subClaimName: sub groupsClaimScope: groups issuer: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7 jwksEndpointUri: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7/v1/keys authorizeBaseEndpointUri: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7/v1/authorize tokenBaseEndpointUri: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7/v1/token refreshToken: true sessionTokenExpiry: 900000 sessionMaxTimeout: 21600000
An example snippet of a Kafka CR configured with MDS and OAuth as the MDS identity provider:
kind: Kafka spec: services: mds: tls: enabled: true tokenKeyPair: secretRef: mds-token provider: oauth: configurations: expectedIssuer: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7 jwksEndpointUri: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7/v1/keys subClaimName: client_id sso: enabled: true clientCredentials: secretRef: oauth-jass-oidc configurations: groupsClaimName: profile_groups subClaimName: sub issuer: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7 jwksEndpointUri: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7/v1/keys authorizeBaseEndpointUri: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7/v1/authorize tokenBaseEndpointUri: https://dev-59009577.okta.com/oauth2/aus96p2og3u7Cpwu65d7/v1/token refreshToken: true
In the Control Center CR, specify OIDC to enable SSO:
kind: ControlCenter spec: dependencies: mds: ssoProtocol: oidc