# Troubleshoot OAuth/OIDC Issues on Confluent Cloud Diagnose and resolve common OAuth/OIDC authentication issues when connecting to Confluent Cloud. ## Debug overview Use this quick checklist before deep-diving into specific errors: - Enable client debug logging for your language and capture logs to a file (see [Enable debug logging](#oauth-debug-logging)). - Check Confluent Cloud audit logs for OAuth authentication events to verify identity pool matching and authentication outcomes (see [Use Confluent Cloud audit logs for OAuth debugging](#oauth-audit-log-debugging)). - Sanity-check client configuration: bootstrap servers, `sasl.mechanism`, token endpoint URL, and OAuth-specific properties (see [Correct Configuration Issues](#oauth-config-issues)). - Retrieve a token with `curl` using the client credentials grant. Distinguish `4xx` (misconfiguration/permission) vs `5xx` (provider/network) responses (see [Token inspection](#oauth-token-inspection)). - Decode the JWT and verify claims: `iss` and `aud` match your pool filter, `exp`/`iat` are valid, clock skew is minimal (preferably < 300 seconds) (see [Token inspection](#oauth-token-inspection)). - Provider specifics: - Microsoft Entra ID: request v2 tokens; use `scope=api:///.default`. - Okta: use the correct authorization server (often `/oauth2/default`). - Network/TLS: validate DNS, connectivity, and certificate chain from the client to the token endpoint (see [Resolve Network and Connectivity Issues](#oauth-network-issues) and [Network diagnostics](#oauth-network-diagnostics)). - Error triage (see [Analyze Errors](#oauth-error-analysis)): - `SASL authentication failed`: check token endpoint URL and credentials. - `JWT_PROCESSING_FAILED` or `KID LOOKUP FAILED`: check issuer/audience, JWKS refresh (see [Manage OAuth-OIDC identity provider configurations on Confluent Cloud](manage-oauth-configs.md#manage-oauth-identity-provider-configs) and [Refresh the JWKS URI](manage-oauth-configs.md#refresh-jwks-uri)), token key rotation cadence, and [verify the JWKS endpoint Content-Type header](#check-jwks-content-type). - `403 Forbidden`: verify RBAC or ACLs and pool filter match. - For what to include in support requests, see [Get help](#oauth-get-help). For OAuth client configuration by language, see [Configure Kafka Clients for OAuth 2.0 Authentication in Confluent Cloud](clients/overview.md#oauth-client-configuration-overview). ## Information to collect for troubleshooting Gather the following artifacts to streamline diagnosis (redact secrets when sharing): - Client configuration: the exact OAuth/OIDC properties used by the client - A representative JWT access token and its decoded claims (issuer, audience, expiry, relevant custom claims) - The identity pool ID and its filter expression, if using identity pools - Recent audit log entries showing OAuth authentication events (see [Use Confluent Cloud audit logs for OAuth debugging](#oauth-audit-log-debugging)) For steps to obtain and inspect tokens and to validate token endpoints, see [Token inspection](#oauth-token-inspection) and [Network diagnostics](#oauth-network-diagnostics). ## Common issues and solutions ## Diagnose and Resolve Common Issues ### Troubleshoot Authentication Failures #### Symptoms - Client fails to connect with authentication errors. - `Invalid credentials` or `Authentication failed` messages. - Connection timeouts during authentication. #### Invalid client credentials Cause : Incorrect client ID or client secret Solution : - Verify client ID and secret match your identity provider configuration. - Check for typos or extra whitespace in credentials. - Ensure credentials are properly URL-encoded if needed. Debug steps : - Enable debug logging to see the exact credentials being sent. - Test credentials directly with your identity provider’s token endpoint. #### Incorrect token endpoint URL Cause : Wrong or inaccessible OAuth token endpoint Solution : - Verify the token endpoint URL is correct and accessible. - Check network connectivity to the endpoint. - Ensure the endpoint supports the `client_credentials` grant type. Debug steps : - Test the endpoint with `curl` or Postman. - Check firewall rules and network access. #### Token expiration issues Cause : Tokens expiring before refresh or invalid expiration handling. Solution : - Implement proper token refresh logic. - Set appropriate buffer time before token expiration. - Monitor token lifecycle and refresh cycles. Debug steps : - Check token expiration times in JWT payload. - Verify refresh logic is working correctly. ### Resolve Network and Connectivity Issues #### Symptoms - Connection timeouts - Network unreachable errors - TLS handshake failures #### Firewall or network restrictions Cause : Network policies blocking OAuth endpoint access Solution : - Configure firewall rules to allow access to OAuth endpoints. - Check corporate proxy settings. - Verify DNS resolution for OAuth domains. Debug steps : - Test connectivity with ping, telnet, or curl. - Check network logs for blocked connections. #### TLS certificate issues Cause : Invalid or expired TLS certificates on OAuth endpoints Solution : - Verify TLS certificate validity. - Update certificate authorities if needed. - Check for certificate chain issues. Debug steps : - Test TLS connection with openssl. - Check certificate expiration dates. #### DNS resolution problems Cause : Unable to resolve OAuth endpoint hostnames Solution : - Verify DNS configuration. - Check for DNS caching issues. - Ensure proper hostname resolution. Debug steps : - Test DNS resolution with nslookup or dig. - Check /etc/hosts file for incorrect entries. ### Correct Configuration Issues #### Symptoms - Configuration parsing errors - Missing required parameters - Invalid parameter values #### Missing required parameters Cause : Incomplete OAuth configuration Solution : - Ensure all required parameters are specified. - Check parameter names and values. - Verify logical cluster ID and identity pool ID. Debug steps : - Review configuration against parameter reference. - Check for typos in parameter names. #### Invalid parameter values Cause : Incorrect parameter formats or values Solution : - Verify parameter value formats. - Check for proper URL encoding. - Ensure boolean values are correct. Debug steps : - Validate parameter values against expected formats. - Check for encoding issues. #### JAAS configuration errors Cause : Malformed JAAS configuration string Solution : - Verify JAAS configuration syntax. - Check for proper escaping of special characters. - Ensure all required JAAS parameters are present. Debug steps : - Parse JAAS configuration manually. - Check for syntax errors. ### Address Identity Provider Issues #### Symptoms - Identity provider authentication failures - Incorrect token claims or scopes - Authorization denied errors #### Incorrect identity provider configuration Cause : Mismatch between client and identity provider settings Solution : - Verify identity provider configuration - Check client registration settings - Ensure proper redirect URIs and scopes Debug steps : - Review identity provider logs - Check client registration details #### Token claim issues Cause : Missing or incorrect JWT claims Solution : - Verify required claims are present in tokens - Check claim values and formats - Ensure proper audience and issuer values Debug steps : - Decode and inspect JWT tokens - Verify claim values against expectations #### Scope and permission issues Cause : Insufficient permissions or incorrect scopes Solution : - Verify required scopes are requested - Check identity provider permission settings - Ensure proper role assignments Debug steps : - Check token scope values - Verify identity provider permissions - For RBAC or ACL configuration guidance, see the cloud APIs for [Role Bindings (iamv2)](../../../../../api.md#cloud-apis) and [ACLs (v3)](../../../../../api.md#cloud-apis). For end-to-end ACL tasks, see [ACL operations](../../../../access-control/acls/operations.md#acl-operations). ## Apply Debugging Techniques ### Enable debug logging #### Java client debug logging ```properties # Kafka client and OAuth debug logging log4j.logger.org.apache.kafka.clients.NetworkClient=DEBUG log4j.logger.org.apache.kafka.common.security.oauthbearer=DEBUG log4j.logger.io.confluent.kafka.clients.oauth=DEBUG ``` #### Python client debug logging ```python import logging # Enable librdkafka debug logging logging.basicConfig(level=logging.DEBUG) # Enable OAuth callback debugging def debug_oauth_callback(args, config): print(f"OAuth callback called with args: {args}") # ... rest of callback implementation ``` #### Go client debug logging ```go // Enable librdkafka debug categories config := &kafka.ConfigMap{ // ... your OAuth config "debug": "security,protocol,broker", } ``` #### JavaScript client debug logging ```javascript // Kafka client const debugConfig = { ...producerConfig, debug: 'security,protocol,broker' }; ``` #### .NET (C#) client debug logging ```csharp // Enable debug logging var config = new ProducerConfig { // ... other config Debug = "all" // Enable all debug categories }; ``` ### Inspect Tokens ## Token inspection ### Decode JWT tokens ```python import jwt import json def inspect_token(token): """Decode and inspect JWT token.""" try: # Decode without verification for inspection decoded = jwt.decode(token, options={"verify_signature": False}) print("Token Header:", json.dumps(decoded['header'], indent=2)) print("Token Payload:", json.dumps(decoded['payload'], indent=2)) return decoded except Exception as e: print(f"Error decoding token: {e}") return None ``` ### Check token expiration ```python import time def check_token_expiration(token_data): """Check if token is expired or near expiration.""" exp = token_data.get('exp') if exp: current_time = int(time.time()) time_until_expiry = exp - current_time print(f"Token expires in {time_until_expiry} seconds") return time_until_expiry > 300 # 5 minute buffer return False ``` ### Run Network Diagnostics #### NOTE If client and server clocks differ significantly (for example, more than 5 minutes), OIDC token validation may fail due to `iat`/`exp` claim checks. Ensure system time is synchronized (for example, via NTP) on client hosts. ## Network diagnostics ### Test OAuth endpoint connectivity ```bash # Test basic connectivity curl -I https://your-oauth-endpoint.com/oauth2/token # Test token endpoint with credentials curl -X POST https://your-oauth-endpoint.com/oauth2/token \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials&client_id=your-client-id&client_secret=your-client-secret" ``` ### Check JWKS endpoint Content-Type header Ensure your JWKS endpoint returns JSON. The response must include either `Content-Type: application/json`, `Content-Type: application/jwk+json`, or `Content-Type: application/jwk-set+json` in the header: ```shell curl -s -D- https://your-idp.example.com/.well-known/jwks.json -o /dev/null | \ grep -i 'content-type:' ``` ### Check TLS certificate ```bash # Check TLS certificate openssl s_client -connect your-oauth-endpoint.com:443 -servername your-oauth-endpoint.com # Check certificate chain openssl s_client -connect your-oauth-endpoint.com:443 -showcerts ``` #### NOTE If you encounter an error such as `error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt (-1)`, verify that the system CA certificate bundle is installed and accessible. On some platforms you may need to set an environment variable to point to the CA bundle (for example, `SSL_CERT_FILE` or `REQUESTS_CA_BUNDLE`) or install/update CA certificates. ### DNS resolution ```bash # Check DNS resolution nslookup your-oauth-endpoint.com dig your-oauth-endpoint.com # Check for DNS issues traceroute your-oauth-endpoint.com ``` ### Analyze Errors ### Common error messages and solutions #### “Invalid credentials” - Check client ID and secret. - Verify identity provider configuration. - Ensure proper URL encoding. #### “Token endpoint not found” - Verify token endpoint URL. - Check network connectivity. - Ensure endpoint supports client_credentials grant. #### “Token expired” - Implement proper token refresh. - Check token expiration handling. - Monitor token lifecycle. - For JWKS refresh behavior and manual refresh steps, see [Manage OAuth-OIDC identity provider configurations on Confluent Cloud](manage-oauth-configs.md#manage-oauth-identity-provider-configs) and [Refresh the JWKS URI](manage-oauth-configs.md#refresh-jwks-uri). #### “Invalid token format” - Verify JWT token structure. - Check token encoding. - Validate token claims. - If your token includes a `cty` header of `JWT`, validation libraries may expect a nested JWT. Either omit `cty` or set it to `JSON` unless you are intentionally using nested tokens. #### “Authorization denied” - Check identity pool configuration. - Verify token claims match pool filters. - Ensure proper permissions. ## Use Confluent Cloud audit logs for OAuth debugging Confluent Cloud audit logs provide valuable insights into OAuth authentication events, helping you diagnose identity pool filter matching issues and authentication failures. Confluent Cloud audit logs capture the authentication outcome but not the raw JWT token contents for security reasons. ### Access Confluent Cloud audit logs To access and consume Confluent Cloud audit logs for OAuth troubleshooting, see [Access and Consume Audit Logs on Confluent Cloud](../../../../../monitoring/audit-logging/configure.md#cloud-activate-audit-logs) for complete setup and consumption instructions. Confluent Cloud automatically enables audit logs for Standard, Enterprise, Dedicated, and Freight clusters. Basic clusters do not include audit logs. These audit logs contain OAuth authentication events in the `confluent-audit-log-events` topic. ### OAuth authentication information in audit logs For OAuth authentication events, audit logs contain the following key information in the `authenticationInfo` section: **Principal and Identity Information:** ```json "authenticationInfo": { "principal": { "identity_pool": { "resource_id": "pool-1" } }, "identity": "crn://confluent.cloud/organization=org-id/identity-provider=op-1/identity=user@example.com", "result": "SUCCESS", "metadata": { "mechanism": "SASL_SSL/OAUTHBEARER", "identifier": "client-identifier" } } ``` **Key fields for troubleshooting:** | Field | Troubleshooting Use | |-----------------------------|------------------------------------------------------------------------------| | `result` | Shows `SUCCESS` or failure status of authentication | | `identity_pool.resource_id` | Indicates which identity pool was matched (if any) | | `identity` | Shows the extracted identity value based on the identity claim configuration | | `metadata.mechanism` | Confirms `SASL_SSL/OAUTHBEARER` for OAuth authentication | | `requestId` | Helps correlate multiple audit events for the same request | When auto pool mapping matches multiple identity pools, Kafka authentication events record a comma-separated list of pool IDs in `principalResourceId`, and Schema Registry and ksqlDB record the full set of mapped pools in the `authorizationInfo.assignedPrincipals` field of their authorization events. For details, see [View mapped identity pools in audit logs](identity-pools.md#auto-pool-mapping-audit-logs). ### Troubleshooting identity pool filter issues **Scenario 1: Authentication fails completely** If you see no OAuth authentication events in audit logs: - Verify your client is reaching Confluent Cloud (check network connectivity) - Confirm OAuth client configuration is correct - Enable client debug logging to see token request/response details **Scenario 2: Authentication succeeds but wrong identity pool** If audit logs show a different `identity_pool.resource_id` than expected: - Review all identity pool filters in your identity provider - Check if multiple pools have overlapping filter criteria - Verify token claims match your intended pool’s filter expression - Consider using more specific filter expressions **Scenario 3: Identity extraction issues** If the `identity` field shows unexpected values: - Check the identity provider’s `identity_claim` configuration - Verify your JWT token contains the expected claim (use [Token inspection](#oauth-token-inspection)) - Ensure the claim value format matches your expectations **Scenario 4: Auto pool mapping not working** For auto pool mapping troubleshooting: - Verify that audit logs show the expected identity pools being matched. - Check that client configuration omits the identity pool ID: - `identityPoolId` SASL extension for Kafka clients : - Java Client: `extension_identityPoolId` in JAAS configuration - librdkafka-based clients version 2.15 or later: `identityPoolId` in the `sasl.oauthbearer.extensions` property > - `bearer.auth.identity.pool.id` property for Schema Registry clients > - `Confluent-Identity-Pool-Id` header for REST requests - For Schema Registry clients, verify you are using `confluent-kafka` version 2.15.0 or later. Earlier versions fail with an error if `bearer.auth.identity.pool.id` is not set. - Confirm multiple pools aren’t conflicting with each other. ### Combining audit logs with token inspection For comprehensive troubleshooting, use audit logs together with manual token inspection: 1. **Check audit logs** to see the authentication outcome and which identity pool was matched 2. **Decode the JWT token** (see [Token inspection](#oauth-token-inspection)) to inspect the actual claims 3. **Compare token claims** with your identity pool filter expressions 4. **Verify claim extraction** by checking if the audit log `identity` field matches the expected claim value from your token Example troubleshooting workflow: 1. **Access audit logs**: Follow [Access and Consume Audit Logs on Confluent Cloud](../../../../../monitoring/audit-logging/configure.md#cloud-activate-audit-logs) to set up audit log consumption 2. **Filter OAuth events**: Search for `SASL_SSL/OAUTHBEARER` mechanism in authentication events 3. **Decode JWT token**: Use [Token inspection](#oauth-token-inspection) to inspect token claims 4. **Compare and correlate**: Match token claims with pool filter expressions and verify the audit log `identity` field matches your expected claim value ### Limitations of audit log debugging Audit logs **do not** contain: - Raw JWT token contents (for security reasons) - Detailed filter evaluation steps - Token validation failure reasons - JWKS validation details For these details, use client debug logging and manual token inspection as described in [Enable debug logging](#oauth-debug-logging) and [Token inspection](#oauth-token-inspection). ## Set up Monitoring and Alerts ### Key metrics to monitor - Authentication success rate - Track successful vs failed authentication attempts - Monitor authentication latency. - Alert on authentication failures. - Token refresh cycles - Monitor token refresh frequency. - Track token expiration times. - Alert on token refresh failures. - Network connectivity - Monitor OAuth endpoint availability. - Track connection latency. - Alert on connectivity issues. - Error rates - Monitor authentication error types. - Track configuration errors. - Alert on increased error rates. ### Recommended alerts - Authentication failure rate > 5% - Token refresh failure rate > 1% - OAuth endpoint response time > 5 seconds - Network connectivity failures - Configuration parsing errors ### Log analysis ```bash # Search for authentication errors grep "authentication failed" kafka-client.log # Search for token refresh issues grep "token refresh" kafka-client.log # Search for network errors grep "connection timeout" kafka-client.log ``` ## Get help If you’re unable to resolve OAuth issues using this troubleshooting guide: 1. **Collect diagnostic information:** - Enable debug logging. - Gather error messages and logs. - Document configuration settings. - Note the exact steps to reproduce the issue. 2. **Check documentation:** - Review identity provider documentation. - Consult Confluent Cloud OAuth documentation. - Check for known issues or limitations. 3. **Contact Confluent Support:** - Open a support request in the [Confluent Support Portal](https://support.confluent.io). - Provide detailed error information. - Include relevant logs and configuration. - Describe the troubleshooting steps already taken. 4. **Community resources:** - Check the [Confluent Community forums](https://forum.confluent.io). - Review issues in the [Confluent GitHub organization](https://github.com/confluentinc). - Consult OAuth/OIDC community resources (for example, [RFC 6749 (OAuth 2.0)](https://www.rfc-editor.org/rfc/rfc6749), [OpenID Connect Core](https://openid.net/specs/openid-connect-core-1_0.html), [OpenID Connect Discovery](https://openid.net/specs/openid-connect-discovery-1_0.html)). ### Information to include in support requests - Client type and version (Java, Python, C#, Go, JavaScript, .NET). - Identity provider type and version, and the token endpoint URL used. - Complete error messages and stack traces. - Relevant client OAuth/OIDC configuration properties (with sensitive data redacted). - Identity pool ID and pool filter expression (if using identity pools). - A representative JWT access token with decoded header and payload (with sensitive values redacted); see [Token inspection](#oauth-token-inspection). - Recent audit log entries showing authentication events (see [Use Confluent Cloud audit logs for OAuth debugging](#oauth-audit-log-debugging)). - Steps to reproduce the issue. - Debug logs and token inspection results. - Network connectivity test results; see [Network diagnostics](#oauth-network-diagnostics).