Configure Authorization for Confluent Platform with Ansible Playbooks

Role-based access control

You can use Ansible Playbooks for Confluent Platform to configure role-based access control (RBAC). The Kafka broker hosts will be configured as the Metadata Service (MDS) hosts.

When enabling RBAC for authorization, you can configure the following Kafka listeners with the authentication options listed:

  • Inter-broker listener: SASL/PLAIN, SASL/SCRAM, SASL/GSSAPI, and mTLS
  • Confluent Platform component listener: SASL/OAUTHBEARER
  • External client listeners: SASL/PLAIN, SASL/SCRAM, SASL/GSSAPI, and mTLS

By default, there are two Kafka listeners: one for inter-broker communication and one for the Confluent Platform components. However, the component listener uses an authentication mode unsupported for external clients, and it is recommended that you configure at least one additional listener for your Kafka clients. See Configure listeners for details.

Do not customize the listener named internal when configuring RBAC.

Requirements

  • An LDAP server reachable from your Kafka broker hosts.
  • Port 8090 must be opened on the Kafka brokers and accessible by all hosts.
  • Set up one principal in LDAP for the MDS admin user to bootstrap roles and permissions for the Confluent Platform component principals. This principal must have the search privilege to read and query users and groups in the LDAP server. It is recommended that you create a user named mds.
  • Set up one principal per Confluent Platform component in your LDAP server. These users are used by the Confluent Platform components to authenticate to MDS and access their respective resources. In the below examples, the following component users are used:
    • Confluent Server: kafka_broker
    • Schema Registry: schema_registry
    • Connect: connect_worker
    • ksqlDB: ksql
    • REST Proxy: rest_proxy
    • Confluent Server REST API: kafka_broker
    • Control Center: control_center
  • (Optional) Generate a key pair to be used by the OAuth-enabled listener as described in Create a PEM key pair.

Note

If using mTLS or Kerberos for inter-broker authentication, you don’t need to set up an LDAP user for Kafka brokers.

Required settings for RBAC

Sample inventory files for RBAC configuration are provided in the directory sample_inventories under the Ansible Playbooks for Confluent Platform home directory.

Add the required variables in your hosts.yml to enable and configure RBAC.

Enable RBAC with Ansible

all:
  vars:
    rbac_enabled: true

Provide LDAP server details for RBAC to look up and validate users

Consult your LDAP admins and see Configure LDAP Group-Based Authorization for MDS and Configure LDAP Authentication for the custom properties you need to set.

The following is an example ldap_config with sample LDAP properties.

kafka_broker:
  vars:
    kafka_broker_custom_properties:
      ldap.java.naming.factory.initial: com.sun.jndi.ldap.LdapCtxFactory
      ldap.com.sun.jndi.ldap.read.timeout: 3000
      ldap.java.naming.provider.url: ldap://ldap1:389
      ldap.java.naming.security.principal: uid=mds,OU=rbac,DC=example,DC=com
      ldap.java.naming.security.credentials: password
      ldap.java.naming.security.authentication: simple
      ldap.user.search.base: OU=rbac,DC=example,DC=com
      ldap.group.search.base: OU=rbac,DC=example,DC=com
      ldap.user.name.attribute: uid
      ldap.user.memberof.attribute.pattern: CN=(.*),OU=rbac,DC=example,DC=com
      ldap.group.name.attribute: cn
      ldap.group.member.attribute.pattern: CN=(.*),OU=rbac,DC=example,DC=com
      ldap.user.object.class: account

If using LDAPS and if the certificate authority is in Kafka’s truststore, include the following properties in the kafka_broker_custom_propertiesdictionary:

kafka_broker:
vars:
  kafka_broker_custom_properties:
    ldap.java.naming.provider.url: ldaps://<ldap-host>:636
    ldap.java.naming.security.protocol: SSL
    ldap.ssl.truststore.location: "{{kafka_broker_truststore_path}}"
    ldap.ssl.truststore.password: "{{kafka_broker_truststore_storepass}}"

If using LDAPs and if the CA is not in Kafka’s truststore, you can make use of the copy file feature:

kafka_broker:
  vars:
    kafka_broker_copy_files:
      - source_path: /path/to/truststore.jks
        destination_path: /var/ssl/private/ldaps.truststore.jks
    kafka_broker_custom_properties:
      ldap.java.naming.provider.url: ldaps://<ldap-host>:636
      ldap.java.naming.security.protocol: SSL
      ldap.ssl.truststore.location: /var/ssl/private/ldaps.truststore.jks
      ldap.ssl.truststore.password: <password>

Provide the super user credentials for bootstrapping RBAC within Confluent Platform

The mds user is used in the inventory file in the sample_inventories directory mentioned above.

all:
  vars:
    mds_super_user: mds
    mds_super_user_password: password

Provide LDAP users for Confluent Platform components

The following users are configured in the inventory file in the sample_inventories directory mentioned above.

all:
  vars:
    kafka_broker_ldap_user: kafka_broker
    kafka_broker_ldap_password: password
    schema_registry_ldap_user: schema_registry
    schema_registry_ldap_password: password
    kafka_connect_ldap_user: connect_worker
    kafka_connect_ldap_password: password
    ksql_ldap_user: ksql
    ksql_ldap_password: password
    kafka_rest_ldap_user: rest_proxy
    kafka_rest_ldap_password: password
    control_center_ldap_user: control_center
    control_center_ldap_password: password

Note

When installing one or more individual Confluent Platform components, use the certificate_authority tag. The tag creates the pem files that MDS requires and copies the files to the managed servers.

ansible-playbook -i hosts.yml confluent.platform.all --tags=certificate_authority

For complete information on installing individual components, see Install individual Confluent Platform components.

Optional settings for RBAC

Add the optional settings in your hosts.yml to enable and configure RBAC.

Provide your own MDS server certificates and key pair for OAuth

all:
  vars:
    create_mds_certs: false

    token_services_public_pem_file: # Path to public.pem
    token_services_private_pem_file: # Path to tokenKeypair.pem

Disable MDS-based ACLs

all:
  vars:
    mds_acls_enabled: false

By default, MDS-based ACLs are enabled when RBAC is enabled.

Enable mutual authentication for RBAC to work with TLS

all:
  vars:
    mds_ssl_mutual_auth_enabled: true

Defaults to the ssl_mutual_auth_enabled setting.

Configure additional principals as RBAC super users

user1 and group1 are used in this example.

all:
  vars:
    rbac_component_additional_system_admins:
      - User:user1
      - Group:group1

Other considerations

Kerberos

When setting sasl_protocol: kerberos, you need keytabs/principals for ZooKeeper, KRaft, Kafka, and external clients. See Configure SASL/GSSAPI (Kerberos) authentication.

However, because the rest of Confluent Platform components use their own listener, there is no need to create Kerberos principals and keytabs for those components. They will authenticate to Kafka using their LDAP user/password.

Role-based access control with centralized MDS

Starting in Confluent Platform 6.0.0, you can use Ansible Playbooks for Confluent Platform to configure role-based access control (RBAC) with a centralized Metadata Service (MDS) on a remote Confluent Platform cluster.

This section provides the additional requirements and settings for the configuration.

The requirements and configuration details in Role-based access control also apply for this configuration.

Requirements

  • The public key used by the MDS for its OAuth-enabled listener must be provided to the current cluster that you are setting RBAC on.
  • The principal used for authenticating to the remote MDS cluster must be a super user on the remote MDS cluster.

Required settings for RBAC with centralized MDS

To enable and configure RBAC with the centralized MDS, add the additional mandatory variables in your inventory file to.

Enable RBAC centralized MDS with Ansible

all:
  vars:
    external_mds_enabled: true

Provide the centralized MDS bootstrap URLs

Specify the URL for the MDS REST API on the Kafka cluster hosting MDS:

all:
  vars:
    mds_bootstrap_server_urls:

For example:

all:
  vars:
    mds_bootstrap_server_urls: https://ip-172-31-34-246.us-east-1.compute.internal:8090,https://ip-172-31-34-246.us-east-2.compute.internal:8090

Provide the centralized MDS bootstrap servers

Specify a list of the hostnames and ports for the listeners hosting the MDS that you wish to connect to: <mds-broker-hostname1>:<port>,<mds-broker-hostname2>:<port>

all:
  vars:
    mds_broker_bootstrap_servers:

For example:

all:
  vars:
    mds_broker_bootstrap_servers: ip-172-31-43-14.us-west-1.compute.internal:9093,ip-172-31-43-14.us-west-2.compute.internal:9093

Provide the centralized MDS broker listener security configuration

Specify the security settings of the remote Kafka broker that the centralized MDS runs on: (mds_broker_bootstrap_servers):

all:
  vars:
    mds_broker_listener:
      ssl_enabled:             ----- [1]
      ssl_mutual_auth_enabled: ----- [2]
      sasl_protocol:           ----- [3]

  • [1] Set ssl_enabled to true if the remote MDS uses TLS.

  • [2] Set ssl_mutual_auth_enabled to true if the remote MDS uses mTLS.

  • [3] Set sasl_protocol to the SASL protocol for the remote MDS. Options are: none, kerberos, sasl_plain, sasl_scram

    The MDS listener must have an authentication mode, mTLS, Kerberos, SASL/PLAIN, or SASL/SCRAM.

    You can set sasl_protocol to none only if ssl_enabled ([1]) and ssl_mutual_auth_enabled ([2]) to true, therefor specifying mTLS authentication mode for the listener.

The following example is for mTLS on the centralized MDS brokers:

all:
  vars:
    mds_broker_listener:
      ssl_enabled: true
      ssl_mutual_auth_enabled: true
      sasl_protocol: none

Provide the paths to the centralized MDS server certificates and key pair for OAuth

all:
  vars:
    create_mds_certs: false
    token_services_public_pem_file:
    token_services_private_pem_file:

Cluster registry

You can use Ansible Playbooks for Confluent Platform to name your clusters within the cluster registries in Confluent Platform.

Cluster registry provides a way to centrally register and identify Kafka clusters in the metadata service (MDS) to simplify the RBAC role binding process and to enable centralized audit logging.

Register the Kafka clusters in the MDS cluster registry using the following variables in the inventory file of the cluster.

  • To register a Kafka cluster in the MDS:

    kafka_broker_cluster_name:

  • To register a Schema Registry cluster in the MDS:

    schema_registry_cluster_name:

  • To register a Kafka Connect cluster in the MDS:

    kafka_connect_cluster_name:

  • To register a ksqlDB cluster in the MDS:

    ksql_cluster_name: