Configure Authentication with Confluent for Kubernetes

Confluent Platform components are configured without authentication by default. This document presents the supported authentication concepts and describes how to configure authentication for Confluent Platform using Confluent for Kubernetes (CFK).

For more details on security concepts in Confluent Platform, see Security.

For a comprehensive tutorial scenario for configuring authentication, see Deploy Secure Confluent Platform.

Authentication overview

Authentication verifies the identity of users and applications connecting to Kafka and other Confluent components.

Authentication to access Kafka

CFK supports the following authentication mechanisms for client applications and Confluent Platform components to access Kafka:

• SASL/PLAIN authentication: Clients use a username/password for authentication. The username/passwords are stored server-side in Kubernetes Secrets.
• mTLS authentication: Clients use TLS certificates for authentication. The certificate has a CN that identifies the client application principal name.

Authentication to access ZooKeeper

CFK supports the following authentication mechanisms for Kafka to access ZooKeeper:

• Digest authentication
• mTLS authentication

Authentication to access Confluent Platform components

CFK supports the following authentication mechanisms for the components, specifically Connect, ksqlDB, Schema Registry, and Confluent Control Center:

• Basic authentication: Username/password for authentication, where the username/passwords are stored in Kubernetes Secrets.
• mTLS authentication: Clients use TLS certificates for authentication.
• LDAP authentication (for Confluent Control Center only): User principals and password credentials are stored in a LDAP server.

Authentication to access MDS

CFK supports the following authentication mechanism for client applications and Confluent Platform components to access Metadata Service (MDS):

• Bearer authentication

Configure authentication to access Kafka

SASL/PLAIN authentication

SASL/PLAIN is a simple username/password mechanism that is typically used with TLS network encryption to implement secure authentication.

The username is used as the authenticated principal, which can then be used in authorization.

Configure SASL/PLAIN as Kafka authentication mechanism

Configure a Kafka listener, in the Kafka custom resource (CR), to use SASL/PLAIN as the authentication mechanism:

spec:
  ...
  listeners:
    external:
      authentication:
        type: plain                 --- [1]
        jaasConfig:                 --- [2]
          secretRef:                --- [3]
        jaasConfigPassThrough:      --- [4]
          secretRef:                --- [5]
          directoryPathInContainer: --- [6]

• [1] Required. Set to plain.

• One of [3], [5], or [6] is required.

• [2] When you use jaasConfig, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates the JAAS config. This is the recommended way to configure SASL/PLAIN for Kafka.

  • [3] Provide a Kubernetes Secret with a list of usernames and passwords.

    The expected key (the file name) is plain-users.json.

    The value for the key (the data in the file) is:

    {
    "username1": "password1",
    "username2": "password2",
    "username3": "password3"
    }
    

• [4] An alternate way to configure JAAS is to use jaasConfigPassThrough. If you have customizations, such as using a custom login handler, you can bypass the CFK automation and provide the configuration directly.

  • [5] Provide a Kubernetes Secret with the expected key and the value.

    The expected key (the file name) is plain-jaas.conf.

    The expected value for the key (the data in the file) is your JAAS config text. See this Confluent Platform doc for understanding JAAS configs. The following example uses the standard login module and specifies two additional users:

    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
       username="admin" \
       password="admin-secret" \
       user_admin="admin-secret" \
       user_user1="user1-secret" \
       user_user2=”user2-secret”;
    

  • [6] Provide the required credential from the directory path in the container.

    The expected key (file name) is plain-jaas.conf. If jaasConfigPassThrough.directoryPathInContainer is configured as /vaults/secrets, the expected file, plain-jaas.conf, exists in the directory path.

    The following example for the expected value for the key (the data in the file) uses the standard login module and specifies two additional users:

    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
       username="admin" \
       password="admin-secret" \
       user_admin="admin-secret" \
       user_user1="user1-secret" \
       user_user2=”user2-secret”;
    

    See CFK Git Hub examples for more information on using the directoryPathInContainer property with Vault.

Configure Confluent components to authenticate to Kafka using SASL/PLAIN

For each of the Confluent components that communicates with Kafka, configure SALS/PLAIN authentication in the component CR as below:

spec:
  ...
  dependencies:
    kafka:
      authentication:
        type: plain                 --- [1]
        jaasConfig:                 --- [2]
          secretRef:                --- [3]
        jaasConfigPassThrough:      --- [4]
          secretRef:                --- [5]
          directoryPathInContainer: --- [6]

• [1] Required. Set to plain.

• One of [2], [3], or [4] is required.

• [2] When you use jaasConfig, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates JAAS config.

  • [3] Provide a Kubernetes Secret with the username/password used to authenticate to Kafka.

    The expected key (filename) is plain.txt.

    The expected value for the key (the data in the file) is:

    username=<username>
    password=<password>
    

• [4] An alternate way to configure JAAS is to use jaasConfigPassThrough. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly.

  • [5] Provide a Kubernetes Secret with the expected key and the value.

    The expected key (filename) is plain-jaas.conf.

    The expected value for the key (the data in the file) is your JAAS config text. See this Confluent Platform doc for understanding JAAS configs. The following example uses the standard login module and specifies two additional users:

    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
       username="admin" \
       password="admin-secret" \
       user_admin="admin-secret" \
       user_user1="user1-secret" \
       user_user2="user2-secret";
    

  • [6] Provide the required credential from the directory path in the container.

    The expected key (file name) is plain-jaas.conf.

    If jaasConfigPassThrough.directoryPathInContainer is configured as /vaults/secrets, the expected file, plain-jaas.conf, exists in the directory path. The following example for the expected value for the key (the data in the file) uses the standard login module and specifies two additional users:

    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
       username="admin" \
       password="admin-secret" \
       user_admin="admin-secret" \
       user_user1="user1-secret" \
       user_user2="user2-secret";
    

    See CFK Git Hub examples for more information on using the directoryPathInContainer property with Vault.

mTLS authentication

Configure mTLS as Kafka authentication mechanism

mTLS utilizes TLS certificates as an authentication mechanism. The certificate provides the identity.

The certificate Common Name (CN) is used as the authenticated principal, which can then be used in authorization.

Configure a Kafka listener as below, in the Kafka CR, to use mTLS as the authentication mechanism:

spec:
  ...
  listeners:
    external:
      authentication:
        type: mtls                                     --- [1]
        principalMappingRules:
        - RULE:.*CN[\s]?=[\s]?([a-zA-Z0-9.]*)?.*/$1/   --- [2]
      tls:
        enabled: true                                  --- [3]

• [1] Required. Set to mTLS.

• [2] Optional. This specifies a mapping rule that extracts the principal name from the certificate Common Name (CN).

  The regular expression (regex) used in the mapping rule is Java mapping API. You can use a site, such as Regular Expression Test Drive, to validate the regex.

• [3] Required for mTLS authentication. Set to true.

Configure Confluent components to authenticate to Kafka using mTLS

For each of the Confluent components that communicates with Kafka, configure the mTLS authentication mechanism in the component CR as below:

spec:
  ...
  dependencies:
    kafka:
      authentication:
        type: mtls               --- [1]
      tls:
        enabled: true            --- [2]

• [1] Required. Set to mtls.
• [2] Required for mTLS authentication. Set to true.

Configure authentication to access ZooKeeper

SASL/DIGEST authentication

Configure DIGEST as ZooKeeper authentication

ZooKeeper supports authentication using the SASL DIGEST-MD5 mechanism.

Enable DIGEST authentication in the ZooKeeper CR as below:

spec:
  ...
  authentication:
    type: digest                 --- [1]
    jaasConfig:                  --- [2]
      secretRef:                 --- [3]
    jaasConfigPassThrough:       --- [4]
      secretRef:                 --- [5]
      directoryPathInContainer:  --- [6]

• [1] Required. Set to digest.

• One of [2], [3], or [4] is required.

• [2] When you use jaasConfig, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates JAAS config. This is the recommended way to configure SASL/DIGEST for ZooKeeper.

  • [3] Provide a Kubernetes Secret with the username/passwords.

    The expected key (the file name) is digest-users.json.

    The expected value for the key (the data in the file) is:

    {
    "username1": "password1",
    "username2": "password2"
    }
    

• [4] An alternate way to configure JAAS is to use jaasConfigPassThrough. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly.

  • [5] Provide a Kubernetes Secret with the expected key and the value. The expected key (the file name) is digest-jaas.conf. The expected value for the key (data in the file) is:

    Server {
      org.apache.zookeeper.server.auth.DigestLoginModule required
        user_super="adminsecret"
        user_user1="user1-secret";
     };
    

  • [6] Provide the required credential from the directory path in the container. The expected key/file name is digest-jaas.conf.

    If jaasConfigPassThrough.directoryPathInContainer is configured as /vaults/secrets, the expected file, digest-jaas.conf, exists in the directory path. The following example for the expected value for the key (the data in the file) uses the standard login module and specifies one additional user:

    Server {
      org.apache.zookeeper.server.auth.DigestLoginModule required
        user_super="adminsecret"
        user_bob="bob-secret";
    };
    

    See CFK Git Hub examples for more information on using the directoryPathInContainer property with Vault.

Configure Kafka to authenticate to ZooKeeper using digest

For Kafka to authenticate to ZooKeeper using digest authentication, configure the Kafka CR as below:

spec:
  ...
  dependencies:
    zookeeper:
      authentication:
        type: digest                --- [1]
        jaasConfig:                 --- [2]
          secretRef:                --- [3]
        jaasConfigPassThrough:      --- [4]
          secretRef:                --- [5]
          directoryPathInContainer: --- [6]

• [1] Required. Set to digest.

• One of [2], [3], or [4] is required.

• [2] When you use jaasConfig, you provide the user names and passwords, and CFK automates configuration. For example, when you add, remove, or update users, CFK automatically updates the JAAS config. This is the recommended way to configure SASL/DIGEST for ZooKeeper.

  • [3] Provide a Kubernetes Secret with the username/password used to authenticate to ZooKeeper.

    The expected key (the file name) is digest.txt.

    The value for the key (the data in the file) is shown below. Double quotes are required around the username and password.

    username="<username>"
    password="<password>"
    

• [4] An alternate way to configure JAAS is to use jaasConfigPassThrough. If you have customizations, such as using custom login handlers, you can bypass the CFK automation and provide the configuration directly.

  • [5] Provide a Kubernetes Secret with the expected key and the value.

    The expected key and filename is digest-jaas.conf.

    The following is an example value (the data in the file) for digest-jaas.conf with a standard login module and a user. The expected value for the key (the data in the file) is:

    Client { //zookeeper dependencies
        org.apache.zookeeper.server.auth.DigestLoginModule required
        username="bob"
        password="password";
    };
    

    See CFK Git Hub examples for more information on using the directoryPathInContainer property with Vault.

mTLS authentication

Configure mTLS as ZooKeeper authentication

Enable mTLS authentication in the ZooKeeper CR as below:

spec:
  ...
  authentication:
    type: mtls

Configure Kafka to authenticate to ZooKeeper using mTLS

For Kafka to authenticate to ZooKeeper using mTLS authentication, configure the Kafka CR as below:

spec:
  ...
  dependencies:
    zookeeper:
      authentication:
        type: mtls               --- [1]
      tls:
        enabled: true            --- [2]

• [1] Required. Set to mtls.
• [2] Required for mTLS authentication. Set to true.

Configure authentication to access Confluent Platform components

Basic authentication

Configure basic authentication for Confluent Platform components

The credentials are stored on the server side, in the following format: <username>: <password>, <role that user is assigned to>

The following roles are supported, by default:

• For ksqlDB: The admin, developer, user, and ksql-user roles are available.
• For Schema Registry: The admin, developer, user, and sr-user roles are available.
• For Confluent Control Center: The Administrators and Restricted roles are available.

Warning

For Connect, there is no support for roles.

Configure basic authentication in the component CR as below:

spec:
  ...
  authentication:
    type: basic                  --- [1]
    basic:
      secretRef:                 --- [2]
      directoryPathInContainer:  --- [3]
      restrictedRoles:           --- [4]
      roles:                     --- [5]

• [1] Required. Set to basic.

• [2] or [3] Required.

• [2] Provide a Kubernetes Secret with the username/passwords.

  The expected key is basic.txt.

  The value for the key is:

  username1: password1,role1
  username2: password2,role2
  

• [3] Provide the required credential for basic authentication. See CFK Git Hub examples for more information on using the directoryPathInContainer property with Vault.

• [4] Optional. A list of restricted roles on the server in Confluent Control Center.

• [5] Optional. A list of roles on the server side.

Configure Confluent components to authenticate to other Confluent Platform component using basic authentication

The username/password for basic authentication are either loaded through the secretRef or through directoryPathInContainer.

Warning

Confluent Control Center does not support connecting to Connect and ksqlDB using basic authentication.

Enable basic authentication in the component CR as below. <component> is the Confluent Platform component that this component needs to authenticate to:

spec:
  ...
  dependencies:
    <component>:
      url:
      authentication:
        type: basic              --- [1]
        basic:
          secretRef:             --- [2]

• [1] Required. Set to basic.

• [2] Required.

  The username and password are loaded through secretRef.

  The expected key is is basic.txt.

  The value for the key is:

  username=<username>
  password=<password>
  

mTLS authentication

Configure mTLS authentication for Confluent Platform components

Configure mTLS authentication in the component CR as below:

spec:
  ...
  authentication:
    type: mtls                --- [1]

• [1] Required. Set to mtls.

Configure Confluent components to authenticate to other Confluent Platform components using mTLS authentication

Enable mTLS authentication in the component CR as below. <component> is the Confluent Platform component that this component needs to authenticate to:

spec:
  ...
  dependencies:
    <component>:
      url:
      authentication:
        type: mtls               --- [1]
      tls:
        enabled: true            --- [2]

• [1] Required. Set to mtls.
• [2] Required for mTLS authentication. Set to true.

LDAP authentication for Confluent Control Center

In addition to basic and mTLS authentication methods, Confluent Control Center supports LDAP as an authentication method.

You can configure the Confluent Control Center CR to pull users and groups from an LDAP server:

kind: ControlCenter
spec:
  authentication:
    type: ldap                                       --- [1]
    ldap:
      secretRef:                                     --- [2]
      roles:                                         --- [3]
      restrictedRoles:                               --- [4]
      property:                                      --- [5]
        #  useLdaps: "true"
        #  contextFactory: "com.sun.jndi.ldap.LdapCtxFactory"
        #  hostname: ""
        #  port: "389"
        #  bindDn: ""
        #  bindPassword: ""
        #  authenticationMethod: ""
        #  forceBindingLogin: "true"
        #  userBaseDn: ""
        #  userRdnAttribute: "sAMAccountName"
        #  userIdAttribute: "sAMAccountName"
        #  userPasswordAttribute: "userPassword"
        #  userObjectClass: "user"
        #  roleBaseDn: ""
        #  roleNameAttribute: "cn"
        #  roleMemberAttribute: "member"
        #  roleObjectClass: "group"

• [1] Required. Set to ldap.

• [2] Secret for LDAP credentials.

  The expected key is ldap.txt.

  The value for the key is:

  username=<username>
  password=<password>
  

• [3] Optional. By default, it’s set to ["Administrators", "Restricted"].

• [4] Optional. List of roles with limited read-only access. No editing or creating allowed in Control Center.

• [5] Required. See Configure LdapLoginModule for details.

Configure authentication to access MDS

Bearer authentication

For each of the Confluent components that communicate with MDS, configure bearer authentication mechanism in the component CR:

spec:
  dependencies:
    mds:
      authentication:
        type: bearer               --- [1]
        bearer:
          secretRef:               --- [2]

• [1] Required. Set to bearer.

• [2] Required.

  The username and password are loaded through secretRef.

  The expected key is bearer.txt.

  The value for the key is:

  username=<username>
  password=<password>
  

When RBAC is enabled (spec.authorization.type: rbac), CFK always uses the Bearer authentication for Confluent components, ignoring the spec.authentication setting. It is not possible to set the component authentication type to mTLS when RBAC is enabled.