Configure Metadata Service (MDS)

Important

This feature is available as a preview feature. A preview feature is a component of Confluent Platform that is being introduced to gain early feedback from developers. This feature can be used for evaluation and non-production testing purposes or to provide feedback to Confluent.

The Metadata Service (MDS) acts as the central authority for all authorization and authentication data. Each Kafka broker in the MDS cluster must be configured with MDS.

Important

The MDS cluster and Kafka cluster can be separate clusters or the same cluster.

Prerequisites
  • Self-managed Confluent Platform for your environment must be downloaded.

  • Active Directory (LDAP service) must be configured. The configurations in this tutorial are based on Active Directory (AD). You must update these configurations to match your LDAP service.

  • Brokers running MDS must be configured with a separate listener for inter-broker communication. Users accessing this listener must be configured with ACLs for authorization. These users may also be configured as super.users if required. But these users cannot rely on access to resources using role-based or group-based access. The broker user must be configured as a super user or granted access using ACLs as described in Authorization using ACLs.

    Brokers accept requests on the inter-listener port before metadata for RBAC authorization has been initialized, but requests on other ports are only accepted after the required metadata, including any LDAP metadata that is available. Broker initialization only completes after all relevant metadata has been obtained and cached. When starting multiple brokers in an MDS cluster with a replication factor of 3 (default) for a metadata topic, at least three brokers must also be started simultaneously to enable initialization to complete on the brokers.

  • REST Proxy services that integrate with AD/LDAP using MDS, will use the user login name as the user principal for authorization decisions. By default, this is also the principal used by brokers for users authenticating using SASL/GSSAPI (Kerberos). If your broker configuration overrides principal.builder.class or sasl.kerberos.principal.to.local.rules to create a different principal, the user principal used by brokers may be different from the principal used by other Confluent Platform components. In this case you should configure ACLs and role bindings for your customized principal for broker resources.

Create a PEM key pair

In this step you create a PEM key pair for use by the token service. This key pair is added to your server.properties file in the next step.

  1. Create the 2048-bit RSA private key This example stores the keys in a folder named /tmp/conf/tokenKeypair.pem.

    mkdir /tmp/conf && openssl genrsa -out /tmp/conf/tokenKeypair.pem 2048
    
  2. Extract public key.

    openssl rsa -in /tmp/conf/tokenKeypair.pem -outform PEM -pubout -out /tmp/conf/public.pem
    

Configure the Kafka Broker to run MDS

Tip

You can store passwords and other configuration data securely by using the confluent secret commands. For more information see Secrets.

  1. Add the following required configuration options to the /etc/kafka/server.properties file. Any content in brackets (<>) must be customized for your environment.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    ############################# Confluent Authorizer Settings  #############################
    authorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer
    confluent.authorizer.access.rule.providers=ACL,RBAC
    super.users=<User:admin;User:mds>
    
    ############################# Identity Provider Settings(LDAP) #############################
    ldap.group.name.attribute=sAMAccountName
    ldap.group.object.class=group
    ldap.group.search.base=CN=Users,DC=rbac,DC=confluent,DC=io
    ldap.java.naming.provider.url=ldap://<hostname>:389
    ldap.java.naming.security.authentication=simple
    ldap.java.naming.security.credentials=<password>
    ldap.java.naming.security.principal=<mds-user-DN>
    ldap.user.name.attribute=sAMAccountName
    ldap.user.object.class=user
    ldap.user.search.base=<user-search-base-DN>
    
    ############################# MDS Server Settings #############################
    confluent.metadata.server.advertised.listeners=http://localhost:8090
    confluent.metadata.server.authentication.roles=**
    confluent.metadata.server.listeners=http://0.0.0.0:8090
    confluent.metadata.server.authentication.method=BASIC
    
    ############################# MDS Token Service Settings #############################
    advertised.listeners=<advertised.listeners>,RBAC://localhost:9092
    confluent.metadata.server.public.key.path=</path/to/public.pem>
    confluent.metadata.server.token.key.path=</path/to/private/tokenKeypair.pem>
    listener.name.rbac.oauthbearer.sasl.login.callback.handler.class=io.confluent.kafka.server.plugins.auth.token.TokenBearerServerLoginCallbackHandler
    listener.name.rbac.oauthbearer.sasl.server.callback.handler.class=io.confluent.kafka.server.plugins.auth.token.TokenBearerValidatorCallbackHandler
    listener.name.rbac.oauthbearer.sasl.jaas.config= \
        org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
            publicKeyPath="/tmp/conf/public.pem";
    listener.name.rbac.sasl.enabled.mechanisms=OAUTHBEARER
    #### Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT#####
    listener.security.protocol.map=<advertised.listeners>,RBAC:SASL_PLAINTEXT
    listeners=<listeners>,RBAC://:9092
    

    For a description of the parameters, see:

  2. Start Confluent Platform.

Configure a Kafka cluster to connect to MDS

Important

If your Kafka cluster is separate from your MDS cluster, you must configure your Kafka cluster to use the MDS cluster.

  1. Add the following MDS configuration to your Kafka properties file (/etc/kafka/server.properties). Any content in brackets (<>) must be customized for your environment.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    ############################# Confluent Authorizer Settings  #############################
    authorizer.class.name=io.confluent.kafka.security.authorizer.ConfluentServerAuthorizer
    confluent.authorizer.access.rule.providers= ACL,RBAC
    
    ############################# MDS Server Settings #############################
    confluent.metadata.bootstrap.servers=<hostname>:9093
    confluent.metadata.security.protocol=SASL_PLAINTEXT
    confluent.metadata.sasl.mechanism=PLAIN
    confluent.metadata.sasl.jaas.config= \
        sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
            username="<broker-username>" \
            password="<broker-password>";
    
    ############################# MDS Token Service Settings #############################
    sasl.mechanism=OAUTHBEARER
    security.protocol=SASL_PLAINTEXT
    sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
        username="<broker-username>" \
        password="<broker-password> " \
    metadataServerUrls="localhost:8090";
    

    For a description of the parameters, see:

  2. Start Confluent Platform.