Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Configure Metadata Service (MDS)

The Metadata Service (MDS) acts as the central authority for all authorization and authentication data. For the Kafka cluster hosting MDS, you must configure MDS on each broker. The MDS can enforce RBAC rules on its host Kafka cluster and across multiple secondary clusters (such as Kafka, Connect, and Schema Registry). So you can use a single Kafka cluster hosting MDS to manage and secure multiple secondary Kafka, Connect, Schema Registry, and ksqlDB clusters. For details about configuring Kafka clusters to run MDS, refer to Configure a primary Kafka cluster to host the MDS and role binding and Configure a secondary Kafka cluster managed by the MDS of the primary Kafka cluster.

Important

The MDS cluster and Kafka cluster can be separate clusters or the same cluster.

Prerequisites

• Self-managed Confluent Platform for your environment must be downloaded.
• Active Directory (LDAP service) must be configured. The configurations in this tutorial are based on Active Directory (AD). You must update these configurations to match your LDAP service.
• Brokers running MDS must be configured with a separate listener for inter-broker communication. To access this listener, a user must be configured with ACLs for authorization. These users may also be configured as super.users if required. But these users cannot rely on access to resources using role-based or group-based access. The broker user must be configured as a super user or granted access using ACLs as described in Authorization using ACLs.
• Brokers will accept requests on the inter-listener port before the metadata for RBAC authorization has been initialized. However, requests on other ports are only accepted after the required metadata has been initialized, including any available LDAP metadata. Broker initialization only completes after all relevant metadata has been obtained and cached. When starting multiple brokers in an MDS cluster with a replication factor of 3 (default) for a metadata topic, at least three brokers must also be started simultaneously to enable initialization to complete on the brokers.
• REST Proxy services that integrate with AD/LDAP using MDS, will use the user login name as the user principal for authorization decisions. By default, this is also the principal used by brokers for users authenticating using SASL/GSSAPI (Kerberos). If your broker configuration overrides principal.builder.class or sasl.kerberos.principal.to.local.rules to create a different principal, the user principal used by brokers may be different from the principal used by other Confluent Platform components. In this case you should configure ACLs and role bindings for your customized principal for broker resources.

Create a PEM key pair

In this step you create a PEM key pair for use by the token service. This key pair is added to your server.properties file in the next step.

1. Create the 2048-bit RSA private key This example stores the keys in a folder named /tmp/conf/tokenKeypair.pem.

   mkdir /tmp/conf && openssl genrsa -out /tmp/conf/tokenKeypair.pem 2048
   

2. Extract public key.

   openssl rsa -in /tmp/conf/tokenKeypair.pem -outform PEM -pubout -out /tmp/conf/public.pem
   

Note

Only use OpenSSL to create the PEM key files. Do not use Keytool; it is not a valid option and will result in an error during startup.

Configure a primary Kafka cluster to host the MDS and role binding

Tip

You can store passwords and other configuration data securely by using the confluent secret commands. For more information see Secrets.

Note

You must complete and run the MDS configuration steps here on all available brokers in your cluster. The configuration example here is for a cluster with a standalone broker. To view an example of a multiple broker (inter-broker) configuration, refer to Configure mTLS Authentication and RBAC for Kafka Brokers.

1. Add the following required configuration options to the /etc/kafka/server.properties file. Any content in brackets (<>) must be customized for your environment.

   ############################# Confluent Authorizer Settings  #############################
   authorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer
   confluent.authorizer.access.rule.providers=ZK_ACL,CONFLUENT
   super.users=<User:admin;User:mds>
   
   ############################# Identity Provider Settings(LDAP) #############################
   ldap.group.name.attribute=sAMAccountName
   ldap.group.object.class=group
   ldap.group.member.attribute=member
   ldap.group.member.attribute.pattern=CN=(.*),DC=rbac,DC=confluent,DC=io
   ldap.group.search.base=CN=Users,DC=rbac,DC=confluent,DC=io
   ldap.java.naming.provider.url=ldap://<hostname>:389
   ldap.java.naming.security.authentication=simple
   ldap.java.naming.security.credentials=<password>
   ldap.java.naming.security.principal=<mds-user-DN>
   ldap.user.name.attribute=sAMAccountName
   ldap.user.object.class=user
   ldap.user.search.base=<user-search-base-DN>
   
   ############################# MDS Server Settings #############################
   confluent.metadata.server.advertised.listeners=http://localhost:8090
   confluent.metadata.server.listeners=http://0.0.0.0:8090
   confluent.metadata.server.authentication.method=BEARER
   
   ############################# MDS Token Service Settings #############################
   advertised.listeners=<advertised.listeners>,RBAC://localhost:9092
   confluent.metadata.server.token.key.path=<path-to-token-key-pair.pem>
   listener.name.rbac.oauthbearer.sasl.login.callback.handler.class=io.confluent.kafka.server.plugins.auth.token.TokenBearerServerLoginCallbackHandler
   listener.name.rbac.oauthbearer.sasl.server.callback.handler.class=io.confluent.kafka.server.plugins.auth.token.TokenBearerValidatorCallbackHandler
   listener.name.rbac.oauthbearer.sasl.jaas.config= \
       org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
       publicKeyPath="<path-to-public.pem>";
   listener.name.rbac.sasl.enabled.mechanisms=OAUTHBEARER
   #### Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT#####
   listener.security.protocol.map=<advertised.listeners>,RBAC:SASL_PLAINTEXT
   listeners=<listeners>,RBAC://:9092
   

   Important

   Do not use token services (listener.name.rbac.sasl.enabled.mechanisms=OAUTHBEARER) for client communication. With RBAC enabled, token services are intended for internal communication between Confluent Platform components only, and not for long-running service principals or client authentication. The OAUTHBEARER setting is for internal use and subject to change, and does not implement a full-featured OAUTH protocol. Therefore, use Kerberos authentication for long-lived or client use cases.

   For a description of the parameters, see:

   • Lines 2-4: Enables RBAC. For more information, see Configuring the Confluent Server Authorizer.

   • Lines 7-18: Configures LDAP so that RBAC can use it. For more information, see Configure LDAP Group-Based Authorization for MDS, Configure LDAP Authentication, and Configuring the LDAP Authorizer.

   • Lines 21-23: Defines listeners and configures HTTPS for MDS. For more information, see Metadata Service Configuration Options.

   • Lines 26-36: Enables MDS to use LDAP; configures MDS to generate JWT Bearer tokens; and configures MDS to talk to other instances of itself. For more information, see Metadata Service Configuration Options.

     Be sure to use the same public keys across components and brokers. Do not use a unique key for each MDS service or component connecting to OAUTH. Use the same private key across the brokers. In this way, MDS can properly decrypt the tokens when doing impersonation. Be aware that after setting up MDS, the OAUTH listener does not display specific errors in relation to the MDS decryption keys (tokenKeypair.pem and public.pem) when they are different.

2. Start Confluent Platform.

Troubleshooting token authentication

Token login and authentication to MDS using OAUTHBEARER can fail without obvious known exceptions or errors appearing in either server.log or metadata-service.log. Such failures may be due to either the public and private key files being corrupt, or because the token was generated using different public and private key files.

To troubleshoot token authentication:

1. Delete both tokenKeypair.pem and public.pem from the folders configured in server.properties and regenerate them. The newly-generated key files will be placed in:

   confluent.metadata.server.token.key.path=<path-to-token-key-pair.pem>
   

2. After regenerating tokenKeypair.pem and public.pem, restart the broker server where MDS is running.

3. On the client machine, delete the local CLI cache (~/.confluent/config.json), which cached the token for the super user after the super user logged in.

4. Log in again using the CLI.

Configure a secondary Kafka cluster managed by the MDS of the primary Kafka cluster

To configure a secondary Kafka cluster managed by the MDS of the primary Kafka cluster:

1. Add the following MDS configuration to your Kafka properties file (/etc/kafka/server.properties). Any content in brackets (<>) must be customized for your environment.

   ############################# Confluent Authorizer Settings  #############################
   authorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer
   confluent.authorizer.access.rule.providers= ZK_ACL,CONFLUENT
   
   ############################# Authorizer Settings #############################
   confluent.metadata.bootstrap.servers=<kafka-with-mds-host-1>:<port>,<kafka-with-mds-host-2>:<port>,...
   confluent.metadata.security.protocol=SASL_PLAINTEXT
   confluent.metadata.sasl.mechanism=PLAIN
   confluent.metadata.sasl.jaas.config= \
       org.apache.kafka.common.security.plain.PlainLoginModule required \
       username="<broker-username>" \
       password="<broker-password>";
   
   ############################# Token-based Listener Settings #############################
   listeners=<listeners>,TOKEN://:9092
   advertised.listeners=<advertised.listeners>,TOKEN://<hostname>:9092
   listener.security.protocol.map=<advertised.listeners>,TOKEN:SASL_PLAINTEXT
   listener.name.token.oauthbearer.sasl.jaas.config= \
       org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
       publicKeyPath="<path-to-public.pem>";
   listener.name.token.oauthbearer.sasl.login.callback.handler.class=io.confluent.kafka.server.plugins.auth.token.TokenBearerServerLoginCallbackHandler
   listener.name.token.oauthbearer.sasl.server.callback.handler.class=io.confluent.kafka.server.plugins.auth.token.TokenBearerValidatorCallbackHandler
   

   For a description of the parameters, see:

   • Lines 2-3: Enables RBAC. For more information, see Metadata Service Configuration Options.
   • Lines 6-12: Configures Kafka so that it can talk to the MDS cluster and “consume” the role bindings. In the example above, SASL_PLAINTEXT/PLAIN is used, but you should use whatever security mechanism is required by the Kafka broker running MDS. Also, you should specify the port exposed by the other broker. For more information, see Metadata Service Configuration Options.
   • Lines 15-22: Enables listener with the OAUTHBEARER SASL mechanism, which is used for impersonation. For more information, see Recommended Kafka Broker configuration.

2. Start Confluent Platform.

Next steps

• Role-Based Access Control Quick Start
• Configure RBAC for Control Center
• ksqlDB and role-based access control (RBAC)
• Configuring Role-Based Access Control for Schema Registry
• Kafka Connect and RBAC
• RBAC REST Proxy security configuration