Authenticate Kafka, KRaft, and MDS
Configure authentication for Apache Kafka®, KRaft, and MDS using Confluent for Kubernetes (CFK). This page covers the supported authentication methods and notes any configuration differences between KRaft and ZooKeeper-based deployments.
By default, CFK configures Kafka, KRaft, and Metadata Service (MDS) without authentication.
For more details on security concepts in Confluent Platform, see Security in Confluent Platform.
For a comprehensive tutorial scenario for configuring authentication, see Deploy Secure Confluent Platform.
Configure authentication to access Kafka and KRaft
This section describes the following methods for the server and client-side Kafka and KRaft authentication:
RBAC and listener authentication precedence
RBAC configuration |
Listener authentication behavior |
|---|---|
role-based access control (RBAC) with LDAP (Lightweight Directory Access Protocol) or OAuth |
CFK adds a token listener using the Metadata Service (MDS). Other Confluent Platform
components connect to Kafka using this token listener
authentication (the authentication method that CFK
automatically configures for the token listener), ignoring the
authentication defined in the component custom resource (CR)
under |
RBAC with mTLS |
CFK does not override the internal listener. Confluent Platform components
connect to Kafka using the internal listener authentication
(the authentication method configured for the internal
listener) defined in the component CR under
|
RBAC with a custom listener instead of the internal listener |
CFK does not override the custom listener authentication (the authentication method configured for the custom listener) and uses it in place of the token listener. The custom listener must support the MDS authentication method for this setup to work. |
SASL/PLAIN authentication
SASL (Simple Authentication and Security Layer) PLAIN is a simple username/password mechanism that is typically used with TLS network encryption to implement secure authentication.
Kafka uses the username as the authenticated principal, which authorization rules can then reference.
Server-side SASL/PLAIN authentication for Kafka and KRaft
Configure the server-side SASL/PLAIN authentication for Kafka and KRaft.
Set up credentials.
You can use the JAAS (Java Authentication and Authorization Service) and JAAS pass-through mechanisms to set up SASL/PLAIN credentials.
In the Kafka and KRaftController custom resources (CRs), configure the listener to use SASL/PLAIN as the authentication mechanism:
kind: <Kafka or kRaftController> spec: listeners: <listener type>: authentication: type: plain --- [1] jaasConfig: --- [2] secretRef: --- [3] jaasConfigPassThrough: --- [4] secretRef: --- [5] directoryPathInContainer: --- [6]
[1] Required. Set to
plain.[2] When you use
jaasConfig, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates the JAAS configuration. This is the recommended way to configure SASL/PLAIN for Kafka.One of [3], [5], or [6] is required. Only specify one.
[3] Provide the name of the Kubernetes secret that you created in the linked section.
[4] If you have customizations, such as using a custom login handler, you can bypass the CFK automation and provide the configuration directly using
jaasConfigPassThrough.[5] Provide a Kubernetes secret that you created in the linked section with the expected key and the value.
[6] Provide the directory path in the container that you set up for the credentials in the linked section.
See CFK GitHub examples for more information on using the
directoryPathInContainerproperty with Vault.
Client-side SASL/PLAIN authentication for Kafka and KRaft
Configure the client-side SASL/PLAIN authentication for other Confluent components to authenticate to Kafka and KRaft.
Set up credentials.
You can use the JAAS and JAAS pass-through mechanisms to set up SASL/PLAIN credentials.
For each of the Confluent components that communicate with Kafka or KRaft, configure SASL/PLAIN authentication in the component CR as follows:
kind: <Confluent component> spec: dependencies: <kafka or kRaftController>: authentication: type: plain --- [1] jaasConfig: --- [2] secretRef: --- [3] jaasConfigPassThrough: --- [4] secretRef: --- [5] directoryPathInContainer: --- [6]
[1] Required. Set to
plain.[2] When you use
jaasConfig, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates JAAS configuration.[3], [5], or [6] is required. Specify only one.
[3] Provide a Kubernetes secret you created in the linked section for this Confluent component to authenticate to Kafka and KRaft.
[4] An alternate way to configure JAAS is to use
jaasConfigPassThrough. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly.[5] Provide a Kubernetes secret that you created in the linked section.
[6] Provide the directory path in the container that you set up in the linked section.
SASL/PLAIN with LDAP authentication
SASL/PLAIN with LDAP callback handler is a variation of SASL/PLAIN. When you use SASL/PLAIN with LDAP for authentication, the username principals and passwords are retrieved from an LDAP server.
You must set up an LDAP server, for example, Active Directory (AD), before configuring and starting up a Kafka cluster with the SASL/PLAIN with LDAP authentication. For more information, see Configuring Kafka Client Authentication with LDAP.
Server-side SASL/PLAIN with LDAP for Kafka and KRaft
Set up credentials.
You can use the JAAS and JAAS pass-through mechanisms to set up SASL/PLAIN credentials.
Create server-side SASL/PLAIN LDAP credentials using JAAS configuration
Create server-side SASL/PLAIN LDAP JAAS pass-through credentials
Note
To implement both a SASL/PLAIN listener and a SASL/PLAIN with LDAP listener in your Kafka cluster, the SASL/PLAIN listener must be configured with
authentication.jaasConfigPassThrough. Configuringauthentication.jaasConfigfor the SASL/PLAIN causes the client’s connection to the SASL/PLAIN with the LDAP listener to fail with ajava.io.EOFException.Configure the listeners in the Kafka or KRaft custom resource (CR):
kind: <Kafka or kRaftController> spec: listeners: <listener type>: authentication: type: ldap --- [1] jaasConfig: --- [2] secretRef: --- [3] jaasConfigPassThrough: --- [4] secretRef: --- [5] directoryPathInContainer:--- [6]
[1] Required for the SASL/PLAIN with LDAP authentication for the specific Kafka or KRaft listener type.
[2] When you use
jaasConfigto pass credentials, you provide the user name and password, and CFK automates configuration. When you add, remove, or update the user, CFK automatically updates the JAAS configuration. This is the recommended way to configure SASL/PLAIN LDAP for Kafka.[3] Provide the name of the Kubernetes secret that you created in the linked section for inter-broker authentication.
[4] An alternate way to configure JAAS is to use
jaasConfigPassThrough. If you have customizations, such as using a custom login handler, you can bypass the CFK automation and provide the configuration directly.[2] [4] To configure authentication type
ldapon external or custom listener, you do not need to specifyjaasConfigorjaasConfigPassThrough.[5] Provide the name of the Kubernetes secret that you created in the linked section for inter-broker authentication.
[6] Provide the directory path in the container that you set up in the linked section.
Configure the identity provider in the Kafka or KRaft CR, and apply the CR with the
kubectl applycommand:kind: <Kafka or kRaftController> spec: identityProvider: --- [1] # type is deprecated --- [2] ldap: --- [3] address: --- [4] authentication: --- [5] type: --- [6] simple: --- [7] tls: enabled: --- [8] configurations: --- [9]
[1] Required for the Kafka authentication type
ldap. Specifies the identity provider configuration.When the MDS is enabled, this property is ignored, and the LDAP configuration in
spec.services.mds.provideris used.[2]
identityProvider.typeis deprecated and ignored if set. The provider used is determined by which sub-block (ldap,oauth,mtls,sso, orfile) is populated.[3] This block includes the same properties used in the
spec.services.mds.provider.ldapblock in this CR.[4] Required. The address of the LDAP server, for example,
ldaps://ldap.confluent.svc.cluster.local:636.[5] Required. The authentication method to access the LDAP server.
[6] Required. Specify
simpleormtls.[7] Required if the authentication type ([6]) is set to
simple.[8] Required if the authentication type ([6]) is set to
mtls. Set totrue.[9] Required. The LDAP configuration settings.
Mixing
FileBasedLoginModule(used by SASL/PLAIN) andPlainLoginModule(used by SASL/PLAIN with LDAP) in the JAAS configs is not supported and can cause the Kafka broker to fail. Therefore, to use both SASL/PLAIN listener and a SASL/PLAIN with LDAP listener in a Kafka cluster, you must configure the SASL/PLAIN listener with JAAS configuration pass-through which enforces the use of thePlainLoginModuleclass.To implement both a SASL/PLAIN listener and a SASL/PLAIN with LDAP listener for a Kafka cluster in KRaft mode, in addition to using JAAS configuration pass-through, you must also provide the config override setting,
listener.name.controller.plain.sasl.jaas.configfor the controller listener in the Kafka CR.An example Kafka CR snippet:
kind: Kafka spec: configOverrides: server: - listener.name.controller.plain.sasl.jaas.config=${file:/mnt/secrets/<internal_listener_SASLPLAIN_secret_name>/plain-jaas.conf:sasl.jaas.config}
See an end-to-end sample scenario in the CFK example repository in GitHub.
An example kRaftController CR:
kind: KRaftController
spec:
identityProvider:
ldap:
address: ldaps://ldap.operator.svc.cluster.local:636
authentication:
simple:
secretRef: credential
type: simple
configurations:
groupMemberAttribute: member
groupMemberAttributePattern: CN=(.*),DC=test,DC=com
groupNameAttribute: cn
groupObjectClass: group
groupSearchBase: dc=test,dc=com
userMemberOfAttributePattern: CN=(.*),DC=test,DC=com
userNameAttribute: cn
userObjectClass: organizationalRole
userSearchBase: dc=test,dc=com
tls:
enabled: true
Client-side SASL/PLAIN with LDAP for Kafka and KRaft
When Kafka is configured with SASL/PLAIN with LDAP, Confluent components and clients authenticate to Kafka as SASL/PLAIN clients. The clients must authenticate as users in LDAP.
See Client-side SASL/PLAIN authentication for Kafka and KRaft for configuration details.
mTLS authentication for Kafka and KRaft
mTLS uses TLS certificates as an authentication mechanism. The certificate provides the identity.
Kafka uses the certificate Common Name (CN) as the authenticated principal, which authorization rules can then reference.
Server-side mTLS authentication for Kafka and KRaft
Starting in CFK 2.10, you can configure and use two authentication methods (mTLS and OAuth or mTLS and LDAP) for Kafka listeners, one with mTLS and another supported method.
Configure the listeners in the Kafka or KRaftController CR as follows, to use mTLS as the authentication mechanism:
kind: Kafka
spec:
listeners:
<listener type>:
authentication:
type: --- [1]
principalMappingRules: --- [2]
- RULE:.*CN[\\s]?=[\\s]?([a-zA-Z0-9.]*)?.*/$1/
mtls: --- [3]
sslClientAuthentication: --- [4]
principalMappingRules: --- [5]
- RULE:.*CN[\\s]?=[\\s]?([a-zA-Z0-9.]*)?.*/$1/
tls:
enabled: true --- [6]
[1] Required.
Set to
mtlsfor setting up mTLS as the only authentication method. The listener authenticates clients only by certificate and does not process OAuth/SASL authentication, even if OAuth settings are present on the listener.To use mTLS together with another supported method, such as OAuth or LDAP, set to that other method, for example
oauth, and keep themtlsblock [3] on the listener. For the full configuration, see OAuth/OIDC and mTLS dual authentication for Kafka listeners.
[2] Optional. This specifies a mapping rule that extracts the principal name from the certificate Common Name (CN) when using mTLS only authentication.
The regular expression (regex) used in the mapping rule is Java mapping API.
Shorthand character classes need to be escaped with another backslash. For example, to use a whitespace (
\s), specify\\s.[3] Required for mTLS authentication in the server-side mTLS authentication mode.
[4] Required. Set to
requiredto enable mandatory mTLS authentication from the client-side and to enforce certificate presentation.Valid values are
requestedandrequired. Userequestedfor optional mTLS.[5] Optional. This specifies a principal mapping rules list from the certificate of the client when using mTLS authentication. See [2] for details.
[6] Required for mTLS authentication. Set to
true.
Client-side mTLS authentication for Kafka and KRaft
For each of the Confluent components that communicate with Kafka or KRaft, configure the mTLS authentication mechanism in the component CR as follows:
kind: <Confluent component>
spec:
dependencies:
<kafka or kRaftController>:
authentication:
type: mtls --- [1]
tls:
enabled: true --- [2]
[1] Required. Set to
mtls.[2] Required for mTLS authentication. Set to
true.
OAuth/OIDC authentication for Kafka and KRaft
Open Authorization (OAuth) 2.0 is an open-standard authorization protocol that gives applications secure, delegated access. Use your own identity provider and centralize identity management across your Confluent Platform and other service deployments on the cloud and on-premises.
Starting with CFK 2.9 and Confluent Platform 7.7, you can configure Confluent components with OAuth/OpenID Connect (OIDC), an OAuth-based authentication mechanism.
For the OAuth overview in Confluent Platform, see OAuth 2.0 for Confluent Platform.
Server-side OAuth/OIDC authentication for Kafka and KRaft
Configure a Kafka or KRaft listener in the Kafka or KRaftController CR as follows to use OAuth/OIDC as the authentication mechanism.
kind: <Kafka or kRaftController>
spec:
listeners:
<listener type>:
authentication:
type: oauth --- [1]
jaasConfig/jaasConfigPassThrough: # Use one
secretRef: --- [2]
oauthSettings:
groupsClaimName: --- [3]
subClaimName: --- [4]
audience: --- [5]
expectedIssuer: --- [6]
jwksEndpointUri: --- [7]
tokenEndpointUri: --- [8]
scope: --- [9]
loginConnectTimeoutMs: --- [10]
loginReadTimeoutMs: --- [11]
loginRetryBackoffMs: --- [12]
loginRetryBackoffMaxMs: --- [13]
tls: --- [14]
[1] Required. Set to
oauth.[2] The secret that contains an OIDC client ID and the client secret for authorization and token request to the identity provider (IdP).
Create the secret that contains two keys with their respective values,
clientIdandclientSecretas following:clientId=<client-id> clientSecret=<client-secret>
[3] Required. The name of the claim in the token for identifying the groups of subject in the JSON Web Tokens (JWT). The default value is
groups.[4] The subject name of the JWT (session token). The default value is
sub. Used in single sign-on (SSO).[5] Required. The intended consumer of the access token. You can specify a comma-delimited list for multiple audiences.
[6] The issuer URL, which is typically the authorization server’s URL. This value is used to compare to the issuer claim in the JWT for verification.
[7] The JSON Web Key Set (JWKS) URI. It is used to verify any JWT issued by the IdP.
[8] The base URI for the token endpoint. This is required for OAuth for inter-broker communication along with
clientIdandclientSecretinjaasConfigorjaasConfigPassThrough.[9] Required only when your identity provider does not have a default scope or your groups claim is linked to a scope.
[10] Connect timeout with IdP in ms.
[11] Read timeout with IdP in ms.
[12] Retry backoff with IdP in ms.
[13] Max retry backoff with IdP in ms.
[14] If the IdP is secured with HTTPS, CFK uses this setting to fetch the OAuth token from the IdP server. Set this setting as described in Configure Network Encryption.
If the Kafka cluster does not have TLS enabled while the IdP server does, you may run into a
tls: failed to verify certificateerror.
OAuth/OIDC and mTLS dual authentication for Kafka listeners
You can use OAuth/OIDC and mTLS together on the same Kafka listener. Set the
listener authentication.type to oauth and add the mtls block with
sslClientAuthentication:
kind: Kafka
spec:
listeners:
<listener type>:
authentication:
type: oauth --- [1]
oauthSettings:
... --- [2]
mtls:
sslClientAuthentication: required --- [3]
tls:
enabled: true
[1] Required. Set to
oauth.If you set
typetomtls, the listener authenticates clients only by certificate and does not process OAuth tokens, regardless of other settings on the listener. Components that connect through such a listener use their own certificate identity, and the roles assigned to that identity apply to all users of the component.[2] The OAuth settings as described in Server-side OAuth/OIDC authentication for Kafka and KRaft.
[3] Required to enable mTLS under OAuth.
For RBAC deployments, also set both provider.mtls and provider.oauth
in the Kafka CR for Metadata Service (MDS). For the RBAC settings, see Enable RBAC for Kafka
and Migrate RBAC to dual authentication with mTLS.
Connect to IdP with self-signed certificates
The current set of SSL properties in Kafka listeners does not let you connect to your IdP with self-signed certificates. You receive an error in the Kafka pod:
[2024-04-17 13:33:53,712] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException:
The OAuth validator configuration encountered an error when initializing the VerificationKeyResolver
Use one of the following workaround options:
Specify a custom trust store through JVM (Java Virtual Machine) arguments in the
spec.configOverrides.jvmsection of the Kafka custom resource:kind: Kafka spec: configOverrides: jvm: - "-Djavax.net.ssl.trustStoreType=JKS" - "-Djavax.net.ssl.trustStore=/mnt/jvmtruststore/truststore.jks" - "-Djavax.net.ssl.trustStorePassword=mystorepassword"
Add the trust store to the Kafka listeners as JAAS configuration, using the parameter
unsecuredLoginStringClaim_sub="thePrincipalName".kind: Kafka spec: configOverrides: server: - listener.name.controller.oauthbearer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId="${file:/mnt/secrets/oauth-jass/oauth.txt:clientId}" clientSecret="${file:/mnt/secrets/oauth-jass/oauth.txt:clientSecret}" refresh_ms="3000" ssl.truststore.location="/mnt/sslcerts/truststore.jks" ssl.truststore.password="mystorepassword" unsecuredLoginStringClaim_sub="thePrincipalName";
Alternatively, you can use a secret to pass the JAAS configuration passthrough information as described in Create server-side SASL/PLAIN JAAS pass-through credentials.
Client-side OAuth/OIDC authentication for Kafka and KRaft
For each of the Confluent components that communicate with Kafka or KRaft, configure the OAuth/OIDC authentication mechanism in the component CR as follows.
kind: <Confluent component>
spec:
dependencies:
<kafka or kRaftController>:
authentication:
type: oauth --- [1]
jaasConfig/jaasConfigPassThrough: # Use one
secretRef: --- [2]
oauthSettings:
tokenEndpointUri: --- [3]
[1] Required. Set to
oauth.[2] The secret that contains an OIDC client ID and the client secret for authorization and token request to IdP.
Create the secret that contains two keys with their respective values,
clientIdandclientSecretas following:clientId=<client-id> clientSecret=<client-secret>
[3] The base URI for the token endpoint.
For the full list of the OAuth settings, see OAuth configuration.
Passwordless OAuth/OIDC authentication with client assertion
A client assertion is a JSON Web Token (JWT) with a collection of information for sharing identity and security information, and it is presented as proof of the client’s identity.
Starting with version 8.0, Confluent Platform supports OAuth client assertion, a secure credential management with passwordless authentication. It uses asymmetric encryption-based authentication, extending Confluent Platform OAuth, and allows you to:
Avoid deploying username and password while securing Confluent Platform.
Streamline and automate client credential rotation on a periodic basis without manual intervention for the client applications.
The following client assertion flows are supported in CFK:
Flow |
Notes |
|---|---|
JSON Web Token (JWT) assertion retrieval from file |
Not recommended for production use cases. Use local client assertion instead. |
Local client assertion |
Recommended for production use cases. |
In CFK 3.0, OAuth client assertion is supported for the following resources:
Day 1 components: Kafka, KRaft, MDS, Schema Registry
Day 2 application resources: KafkaTopic, Kafka REST Class, ConfluentRoleBinding, Schema, SchemaExporter, ClusterLinking
Server-side OAuth client assertion for Kafka and KRaft
Configure a Kafka listener in the Kafka CR as follows to use OAuth client assertion as the authentication mechanism.
To set up client assertion, first, you must complete the server-side OAuth configuration.
For client assertion, configure the additional properties on top of the existing OAuth configurations:
kind: <Kafka or kRaftController>
spec:
listeners:
<listener type>:
authentication:
type: oauth
jaasConfig/jaasConfigPassThrough: # Use one
secretRef:
oauthSettings:
clientAssertion:
clientId: --- [1]
jwtTokenFile: --- [2]
secretRef:
directoryPathInContainer:
localClientAssertion: --- [3]
issuer: --- [4]
subject: --- [5]
audience: --- [6]
tokenLifetime: --- [7]
includeNbfClaim: --- [8]
includeJtiClaim: --- [9]
privateKeyFile: --- [10]
secretRef:
directoryPathInContainer:
encryptedPrivateKey: --- [11]
includeAssertionTemplate:--- [12]
[1] The client ID needed for getting the access token. This field is required only if a clientId is mandatory for your IdP.
For client assertion flow, either
jwtTokenFile([2]) orlocalClientAssertion([3]) must be provided.
To set up JWT assertion retrieval from file, use the following properties:
[2] Required for JWT assertion retrieval from file. The location of the file from which the token (assertion) is read. Use
secretRefordirectoryPathInContainerto specify the value.
To set up local client assertion, use the following properties:
[3] Required for local client assertion.
[4] Required. The name of the claim in the token for identifying the issuer in the client assertion JSON Web Tokens (JWT).
[5] Required. The subject name of the client assertion JWT.
[6] Required. The intended consumer of the client assertion JWT.
[7] Default lifetime for the client assertion. The default is 5 minutes.
[8] Determines whether to include “not before” claim in the client assertion JWT.
[9] Determines whether to include the JWT ID (JTI) claim in the client assertion JWT.
[10] Required. If local assertion creation is required, private key used to sign the client assertion JWT. Use
secretRefordirectoryPathInContainerto provide the key.[11] Required. Set to
trueif the private key is an encrypted PKCS#8 key and if a passphrase is used to decrypt the private key.[12] Set to
trueif the assertion template JSON is provided which contains additional header and payload claims to be included in the client assertion.
Client-side OAuth client assertion for Kafka and KRaft
For each of the Confluent component (currently in Confluent Platform 8.0, Schema Registry only) to authenticate with Kafka or KRaft using OAuth client assertion, configure the client-side OAuth client assertion in the component CR as follows.
For KRaft, the authentication object is under
dependencies.kRaftController.controllerListener.authentication.
To set up client assertion, first, you must complete the client-side OAuth configuration.
For client assertion, configure the additional properties on top of the existing OAuth configurations:
kind: <Confluent component>
spec:
dependencies:
<kafka or kRaftController>:
authentication:
type: oauth --- [1]
oauthSettings:
clientAssertion: --- [2]
[1] Required.
[2] See the client assertion properties for a list of properties you can use.
The following is a sample snippet of Schema Registry to authenticate with Kafka using local client assertion:
kind: SchemaRegistry
spec:
dependencies:
kafka:
bootstrapEndpoint: kafka.operator.svc.cluster.local:9071
authentication:
type: oauth
oauthSettings:
tokenEndpointUri: http://keycloak:8080/realms/sso_test/protocol/openid-connect/token
clientAssertion:
clientId: private-key-client
# jwtTokenFile:
# secretRef: jwt-secret
localClientAssertion:
issuer: private-key-client
subject: private-key-client
audience: http://keycloak:8080/realms/sso_test/protocol/openid-connect/token
privateKeyFile:
secretRef: private-key
includeJtiClaim: true
Configure authentication to access MDS
You can use the following authentication methods to communicate with MDS.
Bearer authentication: You use the bearer token issued by MDS.
OAuth/OIDC authentication: You use the token, provided by your own identity provider and accepted by MDS.
mTLS authentication: You use the principal extracted from the certificate data that the MDS server gets when authenticating a client.
Bearer authentication
Client-side Bearer authentication for MDS
For each of the Confluent components that communicate with MDS, configure the bearer authentication mechanism in the component CR, and then apply the CR with the
kubectl applycommand.kind: <Confluent component> spec: dependencies: mds: authentication: type: bearer --- [1] bearer: secretRef: --- [2] directoryPathInContainer --- [3]
[1] Required. Set to
bearer.[2] or [3] Required. Do not specify both.
[2] To load username and secret, set to the Kubernetes secret you created in the previous section.
[3] Provide the path in the container in the linked section. The expected file,
bearer.txt, must exist in the specified directory path.
OAuth/OIDC authentication for MDS
Server-side OAuth/OIDC authentication for MDS
Configure the MDS in the Kafka CR as follows to use OAuth/OIDC as the authentication mechanism:
kind: Kafka
spec:
services:
mds:
provider:
oidc:
clientCredentials:
secretRef: --- [1]
oauth:
configurations:
groupsClaimName: --- [2]
subClaimName: --- [3]
expectedIssuer: --- [4]
jwksEndpointUri: --- [5]
[1] Required only when enabling SSO. The secret that contains an OIDC client ID and the client secret for authorization and token request to the IdP.
Create the secret that contains two keys with their respective values,
clientIdandclientSecretas following:clientId=<client-id> clientSecret=<client-secret>
[2] Groups in JSON Web Tokens (JWT). The default value is
groups.JWT is issued by Confluent (MDS) to maintain the session. This contains information passed by IdP in id token along with custom claims (if any) added by the MDS.
[3] The subject name of the JWT. The default value is
sub.[4] The issuer URL, which is typically the authorization server’s URL. This value is used to compare to the issuer claim in the JWT for verification.
[5] The JSON Web Key Set (JWKS) URI. It is used to verify any JWT issued by the IdP.
For the full list of the OAuth settings, see OAuth configuration.
Client-side OAuth/OIDC authentication for MDS
For each of the Confluent components that communicate with MDS, configure OAuth/OIDC authentication mechanism in the component CR:
kind: <Confluent component>
spec:
dependencies:
mds:
authentication:
type: oauth --- [1]
oauth:
secretRef: --- [2a]
directoryPathInContainer --- [2b]
configuration:
tokenEndpointUri: --- [3]
[1] Required. Set to
oauth.[2a] or [2b] Specify only one setting.
[2a] The secret that contains the OIDC client ID and the client secret for authorization and token request to the identity provider.
Create the secret that contains two keys with their respective values,
clientIdandclientSecretas following:clientId=<client-id> clientSecret=<client-secret>
[2b] The OIDC client ID and the client secret for authorization and token request to the identity provider (IdP).
For details on providing the credential and required annotations when using Vault, see Provide secrets for Confluent Platform component CR.
[3] The base URI for the token endpoint.
For the full list of the OAuth settings, see OAuth configuration.
Passwordless OAuth/OIDC authentication with client assertion
Starting with version 8.0, Confluent Platform supports OAuth client assertion. For a brief overview of client assertion in CFK, see OAuth client assertion.
Server-side OAuth client assertion for MDS
No client assertion specific configuration is needed for MDS beyond the server-side OAuth configurations.
Client-side OAuth client assertion for MDS
For each of the Confluent components (currently in Confluent Platform 8.0, only for Schema Registry) to authenticate with MDS using OAuth client assertion, configure the client-side OAuth client assertion in the component CR as follows.
To set up client assertion, first, you must complete the client-side OAuth configuration.
For client assertion, configure the additional properties on top of the existing OAuth configurations:
kind: <Confluent component>
spec:
dependencies:
mds:
authentication:
type: oauth --- [1]
oauth:
clientAssertion: --- [2]
[1] Required.
[2] See the client assertion properties for a list of properties you can use.
mTLS authentication for MDS
Server-side mTLS authentication for MDS
Starting in CFK 2.10 and Confluent Platform 7.8, MDS supports mTLS authentication either in single authentication mode or dual authentication mode.
Note
Updating an existing RBAC setup to use mTLS authentication is not currently supported in CFK.
Configure the MDS in the Kafka CR as follows to use the mTLS authentication mechanism:
kind: Kafka
spec:
services:
mds:
provider:
mtls: --- [1]
sslClientAuthentication: --- [2]
principalMappingRules: --- [3]
- ^CN=([a-zA-Z0-9.]*).*$/$1/L
[1] Required.
[2] Required. Set to
requiredto enable mandatory mTLS authentication from the client-side and to enforce certificate presentation.Valid values are
requestedandrequired. Userequestedfor optional mTLS.[3] Mapping rules for extracting principal from the certificate.
Client-side mTLS authentication for MDS
For each of the Confluent components that communicate with MDS, configure the mTLS authentication mechanism in the component CR:
kind: <Confluent component> spec: dependencies: mds: authentication: type: --- [1] sslClientAuthentication: --- [2]
[1] If only using mTLS, set to
mtls.If configuring two authentication methods for MDS, set to
beareroroauth.[2] can be a boolean true or false and enables mTLS authentication from the client side.
Configure the Kafka CR to specify that
kafkaRestconnects with MDS over mTLS.kafkaRestprovides the REST client configuration for the MDS when RBAC is enabled.kind: Kafka spec: dependencies: kafkaRest: authentication: type: mtls sslClientAuthentication: true tls: enabled: true secretRef: tls-kafka
For an example scenario configuration, see the CFK examples repository in GitHub.
File-based authentication
When you set up RBAC only with mTLS authentication, you need to use the file-based authentication provider for Control Center (Legacy) and Confluent CLI.
The file-based provider is not supported when MDS uses LDAP or OAuth authentication.
Server-side file-based authentication for MDS
Create a file with user credentials in the following format:
<username-1>:<password-1> <username-2>:<password-2>
Create a Kubernetes secret that contains file-based username/password you created in the previous step. They secret should have key
userstore.txt. For example:kubectl create secret generic file-secret \ --from-file=userstore.txt=./fileUserPassword.txt \ --namespace confluent
Configure the MDS in the Kafka CR as follows to use the file-based authentication mechanism:
kind: Kafka spec: services: mds: provider: file: --- [1] secretRef: --- [2]
[1] Required.
[2] Create a Kubernetes secret that contains the credential file.
If you need to use a vault to set up file-based authentication for MDS, use the Configuration Override feature. For a sample configuration, see the CFK example repo scenario.
Create authentication credentials for Kafka, KRaft, and MDS
Note
Authentication secret files must use the Linux style line ending that only
uses line feed (\n). The Windows style line ending that uses carriage
return and line feed (\r\n) does not work in Kubernetes secret files.
SASL/PLAIN credentials for Kafka and KRaft
Create server-side SASL/PLAIN credentials using JAAS configuration
When you use jaasConfig to provide required credentials for Kafka and
KRaft, CFK automates configuration. For example, when you add, remove, or
update users, CFK automatically updates the JAAS configuration. This is the
recommended way to configure SASL/PLAIN for Kafka.
The expected key for jaasConfig is plain-users.json.
Create a
.jsonfile and add the expected value, in the following format. This file is used for client authentication.{ "username1": "password1", "username2": "password2", ... "usernameN": "passwordN" }
Create a
.txtfile and add the expected value, in the following format. This file is used for inter-broker authentication.username=<username> password=<password>
Create a Kubernetes secret using the expected keys (
plain-users.jsonandplain-interbroker.txt) and the value files you created in the previous steps.The following example command creates a Kubernetes secret, using the
./creds-kafka-sasl-users.jsonand./creds-kafka-sasl-interbroker.txtfiles that contain the credentials:kubectl create secret generic credential \ --from-file=plain-users.json=./creds-kafka-sasl-users.json \ --from-file=plain-interbroker.txt=./creds-kafka-sasl-interbroker.txt \ --namespace confluent
Create server-side SASL/PLAIN JAAS pass-through credentials
If you have customizations, such as using a custom login handler, you can
bypass the CFK automation and provide the configuration directly using
jaasConfigPassThrough.
The expected key for jaasConfigPassThrough is plain-jaas.conf.
The expected value for the key (the data in the file) is your JAAS configuration text. See PLAIN JAAS configuration overview for understanding JAAS configs.
Create a
.conffile and add the expected value, in the following format.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="<admin username>" \ password="<admin user password>" \ user_admin="<admin user password>" \ user_<additional user1>="<additional user1 password>" \ ... user_<additional userN>="<additional userN password>";
The
usernameandpasswordproperties are used by the broker to initiate connections to other brokers.The
user_<username_N>properties define the passwords for the users,<username_N>, that connect to the broker. The broker validates all client connections, including those from other brokers using these properties.
The following example uses the standard login module and specifies two additional users,
user1anduser2.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="admin-secret" \ user_admin="admin-secret" \ user_user1="user1-secret" \ user_user2="user2-secret";
You can use a Kubernetes secret or a directory path in the container to store the credentials.
Create a Kubernetes secret using the expected key (
plain-jaas.conf) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./creds-kafka-sasl-users.conffile that contains the credentials:kubectl create secret generic credential \ --from-file=plain-jaas.conf=./creds-kafka-sasl-users.conf \ --namespace confluent
Use a directory path in the container to provide the required credentials.
If
jaasConfigPassThrough.directoryPathInContaineris configured as/vaults/secretsin the Kafka CR, the expected file,plain-jaas.conf, must exist in the directory path.For details on providing the credential and required annotations when using Vault, see Provide secrets for Confluent Platform component CR.
See CFK GitHub examples for more information on using the
directoryPathInContainerproperty with Vault.
Create client-side SASL/PLAIN credentials using JAAS configuration
When you use jaasConfig, you provide the user names and passwords, and CFK
automates configuration. For example, when you add, remove, or update users,
CFK automatically updates JAAS configuration.
The expected client-side key for jaasConfig is plain.txt.
Create a
.txtfile and add the expected value, in the following format:username=<username> password=<password>
You specify the name of this value file in the next step when you create a secret.
Create a Kubernetes secret using the expected key (
plain.txt) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./creds-kafka-sasl-users.txtfile that contains the credentials:kubectl create secret generic credential \ --from-file=plain.txt=./creds-kafka-sasl-users.txt \ --namespace confluent
If the user name or password changes in the future, you need to update the credentials in the value file (for example,
./creds-kafka-sasl-users.txt), update the secret for theplain.txtkey, and manually restart the Confluent Platform components that depend on theplain.txtkey. For details, see Update client-side SASL/PLAIN users using JAAS configuration.
Create client-side SASL/PLAIN JAAS pass-through credentials
If you have customizations, such as using a custom login handler, you can bypass
the CFK automation and provide the configuration directly using
jaasConfigPassThrough.
The expected client-side key for jaasConfigPassThrough is
plain-jaas.conf.
Create a
.conffile and add the expected value, in the following format.For example:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="kafka" \ password="kafka-secret";
You specify the name of this value file in the next step when you create a secret.
You can use a Kubernetes secret or a directory path in the container to store the credentials.
Create a Kubernetes secret using the expected key (
plain-jaas.conf) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./creds-kafka-sasl-users.conffile that contains the credentials:kubectl create secret generic credential \ --from-file=plain-jaas.conf=./creds-kafka-sasl-users.conf \ --namespace confluent
Use a directory path in the container to provide the required credentials.
If
jaasConfigPassThrough.directoryPathInContaineris configured as/vaults/secretsin the component CR, the expected file,plain-jaas.conf, must exist in that directory path.For details on providing the credential and required annotations when using Vault, see Provide secrets for Confluent Platform component CR.
SASL/PLAIN LDAP credentials for Kafka and KRaft
Create server-side SASL/PLAIN LDAP credentials using JAAS configuration
The expected server-side key for jaasConfig is plain-interbroker.txt.
Create a
.txtfile and add the expected value, in the following format. You specify the name of this value file in the next step when you create a secret.username=<user> password=<password>
The username and password must belong to a user that exists in LDAP. This is the user that each Kafka broker authenticates when the cluster starts.
Create a Kubernetes secret with the user name and password for inter-broker authentication.
The following example command creates a Kubernetes secret, using the
./creds-kafka-ldap-users.txtfile that contains the credentials:kubectl create secret generic credential \ --from-file=plain-interbroker.txt=./creds-kafka-ldap-users.txt \ --namespace confluent
Create server-side SASL/PLAIN LDAP JAAS pass-through credentials
The expected server-side key for jaasConfigPassThrough is
plain-jaas.conf.
Create a
.conffile and add the expected value. For example:sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="kafka" \ password="kafka-secret";
You specify the name of this value file in the next step when you create a secret.
You can use a Kubernetes secret or a directory path in the container to store the credentials.
Create a Kubernetes secret using the expected key (
plain-jaas.conf) and the value file you created in the previous step.The following example command creates a Kubernetes secret, using the
./creds-kafka-sasl-users.conffile that contains the credentials:kubectl create secret generic credential \ --from-file=plain-jaas.conf=./creds-kafka-sasl-users.conf \ --namespace confluent
Use a directory path in the container to provide the required credentials.
If
jaasConfigPassThrough.directoryPathInContaineris configured as/vaults/secretsin the Kafka CR, the expected file,plain-jaas.conf, must exist in the directory path.For details on providing the credential and required annotations when using Vault, see Provide secrets for Confluent Platform component CR.
Bearer credentials for MDS
Create client-side Bearer credentials for MDS
Provide the required bearer credentials.
The expected key is bearer.txt.
Create a
.txtfile with the expected value in the following format:username=<username> password=<password>
You can use a Kubernetes secret or a directory path in the container to store the bearer credentials.
Create a Kubernetes secret using the expected key (
bearer.txt) and the value.For example, using the
./c3-mds-client.txtfile that you create for Control Center (Legacy) credentials:kubectl create secret generic c3-mds-client \ --from-file=bearer.txt=./c3-mds-client.txt \ --namespace confluent
Use a directory path in the container to provide the required credentials.
For details on providing the credential and required annotations when using Vault, see Provide secrets for Confluent Platform component CR.
See CFK GitHub examples for more information on using the
directoryPathInContainerproperty with Vault.