OAuth/OIDC Service-to-Service Authentication in Confluent Platform¶
Confluent Platform services communicate with each other in various scenarios. You must configure authentication among these services for secure communication. This page describes how to configure OAuth-based service-to-service authentication on your Confluent Platform clusters. For example, OAuth tokens are used to authenticate Schema Registry to Confluent Server brokers, Connect to Schema Registry, Connect to Confluent Server brokers, and between other Confluent Platform components.
This is called the client credentials grant type. Clients use this type to get an access token outside of a user context. Confluent Platform services use the client credentials grant type for authentication among themselves. The grant type used for internal service-to-service authentication is the same grant type used for client-to-Kafka authentication, using the SASL/OAuthBearer mechanism, as specified in KIP-768.
Example of OAuthBearer authentication for Confluent Platform services¶
You must first establish a trust relationships between your Confluent Platform cluster and an OIDC-compliant identity provider (IdP). You then configure IdP endpoints among Confluent Platform services (Confluent Server, Schema Registry, Confluent Control Center, and Connect) to fetch and accept provider-issued OAuth tokens for authentication. To automate the setup of these trust relationships, you can use Confluent for Kubernetes or Ansible Playbooks for Confluent Platform.
The example sequence diagram that follows illustrates a successful authentication flow between two different Confluent Platform services. In this example, assume that Connect is the Confluent Platform Service 1 and Schema Registry is the Confluent Platform Service 2. Further, both these services are configured to fetch and accept OAuth tokens from an OIDC-compliant identity provider already.
Connect (Service 1) wants to register, retrieve schemas to/from Schema Registry. Connect must authenticate and get authorized for the schema request to be successful.
Connect fetches an OAuth token by supplying the IdP with its client credentials and IdP authenticates them. After successful credential validation, the IdP provisions an OAuth token embedding in it the Connect identity.
Connect then reaches out to Schema Registry and provides its OAuth token. The Schema Registry
AuthenticationHandler receives this OAuth token with Connect principal,
scope, claims embedded in it.
The Schema Registry validates the Connect OAuth token using the earlier configured JWKS keys. If successful, the Schema Registry then extracts the principal, scope, claims of Connect. Finally, the Schema Registry looks for access policy rules defined for Connect to access the requested schema and accordingly grants access by establishing a session.
This flow is identical between any Confluent Platform services configured for OAuthBearer authentication. For example, Confluent Server to Schema Registry, Schema Registry to Confluent Server, Connect to Confluent Server, Connect to Schema Registry, Confluent Control Center to Confluent Server, Confluent Control Center to Schema Registry, and so on.
This type of flow is called the client credential grant type of OAuth.
AuthenticationHandler¶
All Confluent Platform services use the AuthenticationHandler to accept OAuth tokens. This
handler is designed to support multiple authentication requests and facilitate
seamless migration from one authentication mechanism to another (for example,
from mTLS to OAuth or from LDAP to OAuth) without causing downtime for clients
during the transition. For more information, see Use Multi-Protocol Authentication in Confluent Platform
and Use the AuthenticationHandler Class for Multi-Protocol Authentication in Confluent Platform.
JWKS keys¶
To validate incoming OAuth token requests, Confluent Platform services must be configured to cache JWKS keys from the corresponding identity provider (IdP). Confluent Platform services do not fetch the keys at the service startup. These services retrieve the keys when the first OAuth verification (OAuth call) happens.
You configure expiry of JWKS keys on the (IdP). If the JWKS keys expire on Confluent Platform due to a rotation policy or due to the expiry, the services immediately fetch updated keys from the IdP to prevent authentication failures.
OAuth/OIDC platform integration using two IdPs¶
Confluent Platform can be configured to accept OAuth/OIDC tokens simultaneously from two different IdPs. One IdP can host all user identities for SSO authentication across Confluent Control Center and Confluent CLI. And the other IdP can host all the service accounts representing the identities of client applications and Confluent Platform services.
Both Confluent for Kubernetes and Ansible Playbooks for Confluent Platform support deployment of OAuth using two different IdPs.
Monitoring authentication¶
It’s useful to get the monitoring data of authentication requests, their
success rates, and associated latencies. You can enable monitoring of
authentication through the AuthenticationHandler. This class generates
metrics for the different Confluent Platform authentication mechanisms, OAuth, mTLS, basic,
and MDS over OAuth.
To enable monitoring, set this property in the Confluent REST Proxy service:
rest.auth.jmx.enabled=true
You should begin to see metrics similar to the following:
Total successful auth requests
Total unsuccessful auth requests
Latency in successful auth request ( sampled )
Latency in un successful auth request (sampled )
Average latency of successful auth requests ( ms )
Max latency of successful auth requests (ms).
99 percentile of latency for successful auth requests (ms)
95 percentile of latency for successful auth requests (ms)
Confluent Platform uses JMX (Java Management Extensions) for exposing metrics from various services, including the Confluent REST Proxy. This allows you to monitor with Confluent Control Center or external tools like JConsole, Prometheus, and Grafana. For more information using these tools with Confluent Platform, see Monitor Consumer Lag in Confluent Platform.