Migrate from LDAP to OAuth RBAC in a RBAC-enabled Confluent Platform Cluster

Starting with Confluent Platform 7.7, you can use OAuth/OIDC-based authentication with RBAC authorization on Confluent Platform clusters. An existing RBAC-enabled Confluent Platform cluster can be migrated from LDAP to OAuth/OIDC authentication. When migrating from LDAP to OAuth/OIDC, get the following benefits:

  • Principal compatibility: There is no impact on the already defined RBAC access rules if principal compatibility is ensured before and after the migration to OAuth/OIDC. For example, if the existing access policy defined for user Bob with principal Bob in the existing cluster, the OAuth token should also mention Bob as the principal for user Bob after migrating to OAuth/OIDC.
  • Group-based authorization: Confluent Platform OAuth/OIDC supports group-based authorization similar to LDAP. As a result, there is no impact for the access policies defined on groups.
  • Zero downtime for clients: During the migration phase, Confluent Platform services can simultaneously accept both LDAP and OAuth tokens for authentication by using the new AuthenticationHandler. Metadata Service (MDS) can also accept both OAuth and LDAP for the backing user store. This ensures there is backward compatibility and zero downtime for clients that are not migrated to OAuth from LDAP. After all the client applications are migrated to OAuth, you can switch the cluster to authenticate with OAuth/OIDC only.

Sample Cluster with Migration from LDAP to OAuth

The following sample migration cluster state diagram shows a Confluent Platform cluster before migrating from HTTP Basic Authentication (LDAP) to OAuth while using RBAC authorization.

Before migrating to OAuth, all authentications run through LDAP.
  • Before migration, the cluster is using HTTP Basic Authentication with LDAP server as the user store at MDS.
  • For Confluent Server brokers, SASL/Plain authentication is used, but powered by LDAP as the user store. After MDS validates the HTTP Basic Authentication credentials of a client using LDAP server, MDS issues a token to the client.
  • The client uses the token in Bearer Auth for subsequent authentication requests across the cluster. Here, the client can be a Confluent Platform service or a producer/consumer application.

Now, in the following diagram, the Confluent Platform cluster is shown during the migration phase where the cluster accepts both LDAP and OAuth from the client applications.

  • Confluent Platform services, like Schema Registry and Connect, are now using AuthenticationHandler, allowing them to accept the previous LDAP-based tokens as well as the new OAuth-based tokens. Additionally, a new SASL/OAuthBearer listener is configured for Confluent Server brokers.
  • Client App2 started using OAuth for authentication while Client App1 continues using LDAP. Also, all internal service-to-service authentications within the cluster are migrated to OAuth.
  • In the example below, no changes were required for RBAC access rules because the principals of the Confluent Platform services, including Client App1 and Client App2, Connect, and Schema Registry, are the same before and after the migration to OAuth.
Migration phase where both LDAP and OAuth are supported on the same cluster.

Steps to migrate LDAP to OAuth in a Confluent Platform cluster

To ensure a smooth transition, review and complete the following process.

  1. Understand the current LDAP RBAC configuration.

    Review your existing LDAP RBAC configuration to understand how roles and permissions are configured.

  2. Configure the Metadata Service (MDS) to support OAuth.

    To modify the Metadata Service (MDS) to enable OAuth support, you need to:

    • Update the confluent.metadata.server.user.store property to LDAP_WITH_OAUTH for a hybrid approach during the migration phase and with OAUTH once all clients are migrated to OAuth.
    • Configure the necessary OAuth endpoints and ensure that the Metadata Service (MDS) can validate OAuth tokens.

    Example configuration of Metadata Service (MDS) for OAuth support

    Similar to your Confluent Server broker configurations, the following settings are required to enable identity provider (IdP)-issued OAuth token validation in MDS. For details on these configurations, see configurations for supporting identity provider tokens in MDS.

    confluent.metadata.server.user.store=LDAP_WITH_OAUTH
confluent.metadata.server.oauthbearer.jwks.endpoint.url=<idp-jwks-endpoint>
confluent.metadata.server.oauthbearer.expected.issuer=<idp-issuer>
confluent.metadata.server.oauthbearer.expected.audience=<configured-audience>
confluent.metadata.server.oauthbearer.sub.claim.name=sub # optional
confluent.metadata.server.oauthbearer.groups.claim.name=groups # optional

  3. Configure your OIDC identity provider to issue OAuth tokens.

    • Set up an OIDC-compliant identity provider (IdP), such as Okta, Keycloak, or another provider.
    • Ensure that your identity provider is configured to issue tokens that Metadata Service (MDS) can validate.

  4. Update your client configurations.

    Update the configurations of clients (for example, producers/consumers, or if using OAuth for the platform service to service authentication, update client configurations for Confluent Server brokers, Schema Registry, REST Proxy, and Connect) to use OAuth for authentication. This involves setting the appropriate OAuth properties in the client configuration files. For details, see Configure Clients for SASL/OAUTHBEARER authentication in Confluent Platform.

    Example of client configuration for OAuth authentication

    To use OAuth authentication with an Confluent Platform cluster, you must configure Kafka clients with the following properties, replacing the placeholders with your actual values:

    sasl.mechanism=OAUTHBEARER
security.protocol=SASL_SSL
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
sasl.login.connect.timeout.ms=15000 # optional
sasl.oauthbearer.token.endpoint.url=<idp-token-endpoint>
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
  clientId="<client-id>" \
  clientSecret="<client-secret>" \
  scope="<scope>"; # optional

  5. Test the configuration.

    Thoroughly test the new OAuth configuration in a staging environment to ensure that authentication and authorization work as expected.

  6. Monitor and validate.

    • Monitor your Confluent Platform cluster after migration to ensure that there are no issues with authentication or authorization.
    • Validate that all users, clients, and services have the correct permissions.

Related content

Confluent for Kubernetes and Ansible Playbooks for Confluent Platform provide full automation support for migrating from LDAP to OAuth in a running RBAC-enabled cluster. See the following for code examples:

To understand how OAuth and RBAC work in Confluent Platform clusters, see Client Flow for OAuth-OIDC using RBAC in Confluent Platform Clusters.

For an overview of OAuth/OIDC support in Confluent Platform, see Use OAuth/OIDC for Authentication in Confluent Platform.