Troubleshoot Confluent Gateway CFK Deployments

Diagnose and resolve common Confluent Gateway issues on Confluent for Kubernetes (CFK). Start with the log-capture and debug-logging steps that follow, then consult Known and common issues for symptom-based resolutions.

Note

The Kubernetes commands on this page assume that you have already set the default namespace. If Confluent Gateway runs outside the default namespace, add -n <namespace> to each command.

Capture Confluent Gateway logs manually

The Confluent Gateway pod logs capture most initialization failures, including network timeouts and credential errors. Inspect them first using the following Kubernetes commands:

# Get the Gateway custom resource
kubectl get Gateway <gateway_name> -o yaml > gateway.yaml

# List Gateway pods
kubectl get pods -l app=confluent-gateway

# View logs from all Gateway pods
kubectl logs -l app=confluent-gateway

# Filter for errors
kubectl logs -l app=confluent-gateway | grep -i error

Review these logs first to identify configuration or connectivity issues before you investigate Confluent Gateway routing or client settings.

Enable debug logging to view stack traces

The DEBUG log level captures detailed logs, including stack traces, to help diagnose complex Confluent Gateway issues.

The GATEWAY_ROOT_LOG_LEVEL environment variable controls the global logging level for Confluent Gateway. Update the podTemplate in your custom resource (CR) to set it to DEBUG, then re-apply the CR:

kind: Gateway
spec:
  podTemplate:
    envVars:
      - name: GATEWAY_ROOT_LOG_LEVEL
        value: DEBUG

Capture Confluent Gateway logs with a support bundle

Use the CFK support bundle command to collect only Confluent Gateway-related diagnostics, including logs, CRs, and related resources:

kubectl confluent support-bundle --only-gateway

Use the generated archive when sharing details with Confluent Support.

Verify client-to-Kafka connectivity

Test a client that connects directly to the backend Apache Kafka® cluster, bypassing Confluent Gateway, to rule out baseline cluster or network issues before troubleshooting Confluent Gateway routing.

Known and common issues

Confluent Gateway pods in CrashLoopBackOff

Symptoms

  • The Confluent Gateway pods restart repeatedly or remain in CrashLoopBackOff.

  • Running the kubectl get pods -l app=confluent-gateway command shows repeated restarts.

  • Pod logs contain configuration parsing errors, license validation failures, or stack traces.

  • Logs report an unrecognized configuration field, such as UnrecognizedPropertyException: Unrecognized field "tls", which indicates a misspelled or misplaced key.

Likely causes

  • Invalid or incomplete Confluent Gateway CR, such as malformed streamingDomains, routes, or secret references.

  • A missing or invalid license secret that GATEWAY_LICENSES references.

Resolution

  1. Inspect the pod logs for configuration and license errors:

    kubectl logs -l app=confluent-gateway
kubectl logs -l app=confluent-gateway | grep -i license

  2. Validate the CR structure:

    • Ensure that the streamingDomains section defines at least one streaming domain and a corresponding bootstrap server.

    • Ensure that the routes section defines at least one valid route.

  3. Verify secret references: confirm that any secretStores and license secrets referenced in the CR exist in the same namespace and contain the expected keys.

  4. Correct the configuration and re-apply the CR:

    kubectl apply -f <gateway-CR-file>

Clients can’t connect to the Confluent Gateway endpoint

Symptoms

  • Client connections to the Confluent Gateway endpoint time out or stall indefinitely during metadata requests.

  • Clients connect successfully when they target the Kafka cluster directly, but fail to connect when routed through Confluent Gateway.

Likely causes

  • The Confluent Gateway pods can’t reach the Kafka bootstrap servers due to DNS, firewall, or network policy issues.

  • Clients connect to broker addresses directly instead of the Confluent Gateway route endpoint.

  • The routes.endpoint value doesn’t match the address published by the Confluent Gateway Kubernetes service.

  • The bootstrapServerId does not match across sections of the CR.

Resolution

  1. Verify that the route endpoint is reachable: confirm that the Confluent Gateway Kubernetes service or load balancer exposes the host and port defined in routes.endpoint.

    kubectl get svc -l app=confluent-gateway

  2. Test Confluent Gateway-to-Kafka connectivity: from a Confluent Gateway pod, test connectivity to the Kafka bootstrap endpoints defined in streamingDomains.kafkaCluster.bootstrapServers[].endpoint. Inspect DNS resolution, firewall rules, and network policies.

  3. Review logs for authentication issues: inspect logs for repeated connection attempts or terminated during authentication messages. If these errors occur, verify your authentication and TLS/SSL configuration settings. For more information, see Configure Security for Confluent Gateway using Confluent for Kubernetes.

  4. Verify broker endpoint resolution: when you use the host-based broker identification strategy, where routes.brokerIdentificationStrategy.type is host, clients must resolve individual broker addresses to the Confluent Gateway endpoint. If your broker address pattern is broker-$(nodeId).gateway.example.com, ensure that your DNS provider routes these subdomains, including a wildcard record, to the Confluent Gateway service.

Protocol or TLS/SSL mismatch

Symptoms

  • Client logs or Confluent Gateway logs show errors such as frame size exceptions (for example, FrameOversizedException) or TLS handshake failures.

  • Clients use SASL_SSL or SSL, but the route uses plaintext, or vice versa.

Likely causes

  • The client’s security.protocol doesn’t match the configured security for the route.

  • TLS-encrypted traffic reaches a plaintext route endpoint, or plaintext traffic reaches an SSL-only route.

Resolution

  1. Confirm the route configuration: inspect the routes[].security.client.tls settings in the Confluent Gateway CR.

    • If you omit security.client.tls, the route expects plaintext traffic.

    • If you define security.client.tls, configure the client to use an SSL protocol, such as SASL_SSL or SSL.

  2. Align the client configuration:

    • For unencrypted connections, set the client protocol to PLAINTEXT.

    • When using SASL authentication without TLS, set the client protocol to SASL_PLAINTEXT.

    • For TLS routing, set the client protocol to SSL or SASL_SSL and configure the truststore and keystore as required.

  3. Re-apply the CR if you change route security, and confirm that clients can connect again.

Authentication swap not behaving as expected

Symptoms

  • Clients using authentication swap fail to authenticate with the backend cluster.

  • Logs report errors related to secret store lookups, Java Authentication and Authorization Service (JAAS) configuration failures, or callback handler initialization.

  • Backend brokers report invalid credentials despite the client providing valid local credentials.

Likely causes

  • Route security isn’t configured with auth: swap.

  • The secretStore referenced in the routes section is incorrect or missing.

  • The secret store provider type is incorrect, or the credentials required to access the store are incorrect.

Resolution

  1. Verify the route swap configuration: ensure that the route performs authentication swapping.

    kind: Gateway
spec:
  routes:
    - name: <route_name>
      security:
        auth: swap

  2. Avoid hard-coded broker credentials: don’t embed static broker credentials in the CR. For swap scenarios, Confluent Gateway must resolve credentials dynamically from a configured secret store.

  3. Validate your secret store configuration:

    • Provider types are case-sensitive. Verify that you use the exact casing required. For example, use Azure, not AZURE.

    • Confirm that the Confluent Gateway has the necessary permissions to read secrets from the provider and can reach the provider’s API endpoint over the network.

    To learn more about secret store configuration, see Configure Security for Confluent Gateway using Confluent for Kubernetes.

  4. Restart the Confluent Gateway pods so the updated authentication settings take effect:

    kubectl delete pod -l app=confluent-gateway

SASL/OAUTHBEARER passthrough fails after a failover

Symptoms

Clients that use SASL/OAUTHBEARER identity passthrough to Confluent Cloud fail to authenticate after a failover.

Likely causes

Clients that use SASL/OAUTHBEARER identity passthrough send a logical cluster (lkc) extension to identify the target cluster. After a failover, the active cluster changes, so the client-supplied value becomes invalid and authentication fails.

Resolution

Configure passthroughConfig on the route so that Confluent Gateway injects the correct logical cluster extension on behalf of the clients. This keeps client configurations independent of the active cluster.

For more information, see Configure identity passthrough.

License and route limit issues

Symptoms

  • The Confluent Gateway fails to initialize additional routes.

  • Logs report warnings on license or route-limit exhaustion.

Likely causes

Trial mode limits Confluent Gateway to a maximum of four routes. To configure more routes, you need an Enterprise license.

Resolution

  1. Apply an Enterprise license: Trial mode allows a maximum of four routes, and Enterprise mode supports unlimited routes. To create an Enterprise license secret, reference it from the CR, and review the license modes, see Configure Confluent Gateway license.

  2. Inspect the license status: after re-applying the CR, confirm that the Confluent Gateway recognizes the license.

    kubectl logs -l app=confluent-gateway | grep -i license