Configure OAuth Authentication using SASL/OAUTHBEARER in Confluent Platform¶

Note

Early Access Program for OAuth Support

OAuth support for Confluent Platform is currently available in an Early Access Program. An Early Access Program feature is a component of Confluent Platform introduced to gain feedback. This feature should be used only for evaluation and non-production testing purposes or to provide feedback to Confluent.

Early Access Program features are intended for evaluation use in development and testing environments only, and not for production use. The warranty and Support Services provisions of your agreement with Confluent do not apply to Early Access Program features. Confluent may discontinue providing preview releases of the Early Access Program features at any time at Confluent’s sole discretion.

Overview¶

OAuth 2.0 support for Confluent Platform lets you “Bring Your Own Identity” to Confluent Platform, leveraging your identity provider (IdP) to authenticate users and client applications. OAuth 2.0 provides a more secure and convenient way to grant access to your Confluent Cloud resources for your users and client applicaitions. OAuth 2.0 is an open standard for authorization that enables third-party applications to obtain limited access to web services. Auth 2.0 also provides authorization flows for web and desktop applications, and mobile devices.

Confluent Server OAuth support¶

OAuth support for Confluent Platform provides the following benefits:

LDAP independence for Confluent Server RBAC
OAuth support for MDS REST APIs. If you have OAuth-OIDC JWTs, you can use the MDS REST APIs directly for administrative operations using OAuth.
Support for migration and upgrade scenarios to enable client authentication and authorization with both OAuth as well as LDAP/mTLS, etc.
Token-based group management for RBAC where OAuth-OIDC JWTs serve as the source of groups.
Client applications, like producers and consumers, can use OAuth-OIDC JWTs to authenticate and authorize to the Confluent Server.
In addition to the OAuth support described in KIP-768, Confluent Platform supports OAuth for enterprise features, like RBAC and group authorization.
You can Bring Your Own Identity (IdP) to Confluent Server with OAuth support.

Limitations¶

The following limitations apply to OAuth support for Confluent Platform during the Early Access Program:

OAuth support for Confluent Platform is available in Confluent Platform 7.6 as an Early Access feature. that provides a minimal Confluent Platform configuration (consisting of a Kafka broker and MDS), which lets you build applications that effortlessly produce and consume data through Kafka brokers, with RBAC authorization managed by MDS.
Ansible Playbooks for Confluent Platform and Confluent for Kubernetes orchestration is currently available in Confluent Platform 7.6 Early Access using configuration override support to deploy clusters with OAuth authentication for client applications and authorization using ACLs.

Configure Kafka brokers¶

To use OAuth authentication with Confluent Platform, you must configure Kafka brokers with a SASL/OAUTHBEARER listener.

You can use the OIDC discovery endpoint to get the values for your IdP’s JWKS URI <idp-jwks-endpoint>, token endpoint (<idp-token-endpoint>), and other values. Typically, the OIDC discovery endpoint is located at https://<idp-domain>/.well-known/openid-configuration.
Verify if the claim value for groups is compatible with the token configuration in your identity provider.

The following example shows how to configure Kafka brokers with a SASL/OAUTHBEARER listener. Replace the values in angle brackets with the values for your IdP.

# Configuring a new listener
listeners=..,EXTERNAL://:9095
advertised.listeners=..,EXTERNAL://<host>:9095
listener.security.protocol.map=...,EXTERNAL:SASL_PLAINTEXT

# Adding OAuth to the Kafka listener
listener.name.external.sasl.enabled.mechanisms=OAUTHBEARER
listener.name.external.sasl.oauthbearer.jwks.endpoint.url=https://my-good-idp.com/oauth2/keys
listener.name.external.sasl.oauthbearer.expected.audience=api://default
listener.name.external.oauthbearer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
listener.name.external.oauthbearer.sasl.server.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler
listener.name.external.principal.builder.class=io.confluent.kafka.security.authenticator.OAuthKafkaPrincipalBuilder
confluent.oauth.groups.claim.name=groups

Optionally, you can use the SASL/OAUTHBEARER listener as the interbroker listener, using the following configuration:

inter.broker.listener.name=EXTERNAL
sasl.mechanism.inter.broker.protocol=OAUTHBEARER
listener.name.external.sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
listener.name.external.sasl.oauthbearer.token.endpoint.url=<idp-token-endpoint>
listener.name.external.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
      clientId="<client-id>" \
      clientSecret="<client-secret>"
      scope="<scope>"; # scope is optional

When you use SASL/OAUTHBEARER as the interbroker listener, ensure that the principal in the authentication token of the client serves as the superuser. To do this, add the following configuration:

super.users= ...;User:<sub_claim_value>   # usually User:<client-id>

Configure for KRaft mode¶

When you use Confluent Platform in KRaft mode, you must add the following configuration to the Kafka controller for group-based authorization to work:

listener.name.external.principal.builder.class=io.confluent.kafka.security.authenticator.OAuthKafkaPrincipalBuilder
confluent.oauth.groups.claim.name=groups

Configure Kafka clients¶

To use OAuth authentication with Confluent Platform, you must configure Kafka clients with the following properties:

sasl.mechanism=OAUTHBEARER
security.protocol=SASL_PLAINTEXT
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
sasl.login.connect.timeout.ms=15000 # optional
sasl.oauthbearer.token.endpoint.url=<idp-token-endpoint>
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
  clientId="<client-id>" \
  clientSecret="<client-secret>"
  scope="<scope>"; # optional

Scope is optional, but is required when your identity provider does not have a default scope or your groups claim is linked to a scope.

Configure Kafka console consumer clients¶

Client configurations are passed in the client properties file. To use OAuth authentication with Kafka console consumer clients, you must add the following properties to your clients:

./bin/kafka-console-consumer \
  --bootstrap-server <host>:9095 \
  --topic purchases \
  --consumer.config oauth-client.properties

Configure MDS for OAuth support¶

Similar to your Kafka broker configurations, the following settings are required to enable IdP-issued OAuth token validation in MDS. For details on these configurations, see configurations for supporting IdP tokens in MDS.

confluent.metadata.server.user.store=OAUTH
confluent.metadata.server.oauthbearer.jwks.endpoint.url=<idp-jwks-endpoint>
confluent.metadata.server.oauthbearer.expected.issuer=<idp-issuer>
confluent.metadata.server.oauthbearer.expected.audience=Confluent,api://default,https://my-company.com
confluent.metadata.server.oauthbearer.sub.claim.name=sub
confluent.metadata.server.oauthbearer.groups.claim.name=groups

Group claims are included by default — you need to verify if the claim value for groups is compatible with the token configuration in your identity provider.

Configure MDS for TLS¶

If your identity provider uses a self-signed certificate and not one from a CA authority, you must add the identity provider’s TLS certificate to the MDS truststore. To reload the truststore with the TLS certificate, you must restart Confluent Server (confluent-server). For details, .. see Add the identity provider TLS certificates to the MDS truststore.

For development environments, you can use http (instead of https) to connect to your identity provider. For production environments, you should use an https URL in your configurations.

Configure MDS to enable OAuth without LDAP¶

Basic authentication (username:password) is not supported in this case. To interact with MDS requires a token from your identity provider.

confluent.metadata.server.user.store=OAUTH

Configure MDS to enable OAuth with LDAP¶

You can configure MDS to enable OAuth support without removing LDAP. This is mainly useful for client migration scenarios, and should be removed after migration.

confluent.metadata.server.user.store=LDAP_WITH_OAUTH

Add groups in identity providers¶

Depending on your identity provider, you might need to add groups to the token claims.

Applications cannot be added to user group in Okta. However, Okta provides an application profile that you can use as an identity attribute in an access token.

This operation cannot be performed using the Okta UI, but is available using their APIs. Here is a sample cURL request to create an application:

curl --location 'https://my-org.okta.com/api/v1/apps' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'Authorization: SSWS <token>' \
--data '{
      "name": "oidc_client",
      "label": "Sales Producer Application",
      "signOnMode": "OPENID_CONNECT",
      "credentials": {
        "oauthClient": {
          "token_endpoint_auth_method": "client_secret_post"
        }
      },
      "profile": {
        "groups": ["ads", "fb-ads", "marketing-team"]
        },
      "settings": {
        "oauthClient": {
          "client_uri": null,
          "logo_uri": null,
          "response_types": [
            "token"
          ],
          "grant_types": [
            "client_credentials"
          ],
          "application_type": "service",
          "consent_method": "REQUIRED",
          "issuer_mode": "ORG_URL"
        }
      }
    }'

In the above example, the profile attribute is used to add groups to the token claims. The groups attribute is an array of groups that you want to add to the token claims. In this example, three groups called “ads”, “fb-ads”, and “marketing-team” are added to profile of “Sales Producer Application”. For more information, see Add application in the Okta documentation.

To configure the claim in the Okta UI, go to Security → API → <your-authorization-server> → Claims and enter the claim name (groups) and your own expression for the value (for example, app.profile.groups).

A group can be assigned to a client/service application in Keycloak identity provider. To add a group, select your client from the clients list, add it to user groups and add a corresponding token mapper.

To add a client application to a user group in the Keycloak UI,

Go to Clients > Client details > Service accounts roles and click the link in the information message.
Add a token mapper to get the groups in the token. Go to Clients > Client details > Client scopes > Add Mapper > By Configuration and click Create.

A custom claim groups can be added to the token of client_app1. You can choose the name of the claim based on your use and configuration. You should keep same claim name for human user’s ID token (for SSO) as well as the client application.