Troubleshoot Confluent Gateway CFK Deployments

Diagnose and resolve common Confluent Gateway issues on Confluent for Kubernetes (CFK). Start with the log-capture and debug-logging steps that follow, then consult Known and common issues for symptom-based resolutions.

Note

The Kubernetes commands on this page assume that you have already set the default namespace. If Confluent Gateway runs outside the default namespace, add -n <namespace> to each command.

Capture Confluent Gateway logs manually

The Confluent Gateway pod logs capture most initialization failures, including network timeouts and credential errors. Inspect them first using the following Kubernetes commands:

# Get the Gateway custom resource
kubectl get Gateway <gateway_name> -o yaml > gateway.yaml

# List Gateway pods
kubectl get pods -l app=<gateway-name>

# View logs from all Gateway pods
kubectl logs -l app=<gateway-name>

# Filter for errors
kubectl logs -l app=<gateway-name> | grep -i error

Review these logs first to identify configuration or connectivity issues before you investigate Confluent Gateway routing or client settings.

Enable debug logging to view stack traces

The DEBUG log level captures detailed logs, including stack traces, to help diagnose complex Confluent Gateway issues.

The GATEWAY_ROOT_LOG_LEVEL environment variable controls the global logging level for Confluent Gateway. Update the podTemplate in your custom resource (CR) to set it to DEBUG, then re-apply the CR:

kind: Gateway
spec:
  podTemplate:
    envVars:
      - name: GATEWAY_ROOT_LOG_LEVEL
        value: DEBUG

Capture Confluent Gateway logs with a support bundle

Use the CFK support bundle command to collect only Confluent Gateway-related diagnostics, including logs, CRs, and related resources:

kubectl confluent support-bundle --only-gateway

Use the generated archive when sharing details with Confluent Support.

Verify client-to-Kafka connectivity

Test a client that connects directly to the backend Apache Kafka® cluster, bypassing Confluent Gateway, to rule out baseline cluster or network issues before troubleshooting Confluent Gateway routing.

Known and common issues

Confluent Gateway pods in CrashLoopBackOff

Symptoms

The Confluent Gateway pods restart repeatedly or remain in CrashLoopBackOff.
Running the kubectl get pods -l app=<gateway-name> command shows repeated restarts.
Pod logs contain configuration parsing errors, license validation failures, or stack traces.
Logs report an unrecognized configuration field, such as UnrecognizedPropertyException: Unrecognized field "tls", which indicates a misspelled or misplaced key.

Likely causes

Invalid or incomplete Confluent Gateway CR, such as malformed streamingDomains, routes, or secret references.
A missing or invalid license secret that GATEWAY_LICENSES references.

Resolution

Inspect the pod logs for configuration and license errors:

kubectl logs -l app=<gateway-name>
kubectl logs -l app=<gateway-name> | grep -i license

Validate the CR structure:
- Ensure that the streamingDomains section defines at least one streaming domain and a corresponding bootstrap server.
- Ensure that the routes section defines at least one valid route.
Verify secret references: confirm that any secretStores and license secrets referenced in the CR exist in the same namespace and contain the expected keys.
Correct the configuration and re-apply the CR:
```
kubectl apply -f <gateway-CR-file>
```

Clients can’t connect to the Confluent Gateway endpoint

Symptoms

Client connections to the Confluent Gateway endpoint time out or stall indefinitely during metadata requests.
Clients connect successfully when they target the Kafka cluster directly, but fail to connect when routed through Confluent Gateway.

Likely causes

The Confluent Gateway pods can’t reach the Kafka bootstrap servers due to DNS, firewall, or network policy issues.
Clients connect to broker addresses directly instead of the Confluent Gateway route endpoint.
The routes.endpoint value doesn’t match the address published by the Confluent Gateway Kubernetes service.
The bootstrapServerId does not match across sections of the CR.

Resolution

Verify that the route endpoint is reachable: confirm that the Confluent Gateway Kubernetes service or load balancer exposes the host and port defined in routes.endpoint.
```
kubectl get svc | grep <gateway-name>
```
Test Confluent Gateway-to-Kafka connectivity: from a Confluent Gateway pod, test connectivity to the Kafka bootstrap endpoints defined in streamingDomains.kafkaCluster.bootstrapServers[].endpoint. Inspect DNS resolution, firewall rules, and network policies.
Review logs for authentication issues: inspect logs for repeated connection attempts or terminated during authentication messages. If these errors occur, verify your authentication and TLS/SSL configuration settings. For more information, see Configure Security for Confluent Gateway using Confluent for Kubernetes.
Verify broker endpoint resolution: when you use the host-based broker identification strategy, where routes.brokerIdentificationStrategy.type is host, clients must resolve individual broker addresses to the Confluent Gateway endpoint. If your broker address pattern is broker-$(nodeId).gateway.example.com, ensure that your DNS provider routes these subdomains, including a wildcard record, to the Confluent Gateway service.

Protocol or TLS/SSL mismatch

Symptoms

Client logs or Confluent Gateway logs show errors such as frame size exceptions (for example, FrameOversizedException) or TLS handshake failures.
Clients use SASL_SSL or SSL, but the route uses plaintext, or vice versa.

Likely causes

The client’s security.protocol doesn’t match the configured security for the route.
TLS-encrypted traffic reaches a plaintext route endpoint, or plaintext traffic reaches an SSL-only route.

Resolution

Confirm the route configuration: inspect the routes[].security.client.tls settings in the Confluent Gateway CR.
- If you omit security.client.tls, the route expects plaintext traffic.
- If you define security.client.tls, configure the client to use an SSL protocol, such as SASL_SSL or SSL.
Align the client configuration:
- For unencrypted connections, set the client protocol to PLAINTEXT.
- When using SASL authentication without TLS, set the client protocol to SASL_PLAINTEXT.
- For TLS routing, set the client protocol to SSL or SASL_SSL and configure the truststore and keystore as required.
Re-apply the CR if you change route security, and confirm that clients can connect again.

Authentication swap not behaving as expected

Symptoms

Clients using authentication swap fail to authenticate with the backend cluster.
Logs report errors related to secret store lookups, Java Authentication and Authorization Service (JAAS) configuration failures, or callback handler initialization.
Backend brokers report invalid credentials despite the client providing valid local credentials.

Likely causes

The security.cluster entry for the streaming domain isn’t configured with auth: swap. A streaming domain with no entry uses identity passthrough.
The secretStore referenced in the security.cluster entry is incorrect or missing.
The secret store provider type is incorrect, or the credentials required to access the store are incorrect.

Resolution

Verify the route swap configuration: ensure that the route performs authentication swapping.

kind: Gateway
spec:
  routes:
    - name: <route_name>
      security:
        cluster:
          <streaming_domain_name>:
            auth: swap
            secretStore: <secret_store_name>

Avoid hard-coded broker credentials: don’t embed static broker credentials in the CR. For swap scenarios, Confluent Gateway must resolve credentials dynamically from a configured secret store.
Validate your secret store configuration:
- Provider types are case-sensitive. Verify that you use the exact casing required. For example, use Azure, not AZURE.
- Confirm that the Confluent Gateway has the necessary permissions to read secrets from the provider and can reach the provider’s API endpoint over the network.
To learn more about secret store configuration, see Configure Security for Confluent Gateway using Confluent for Kubernetes.
Restart the Confluent Gateway pods so the updated authentication settings take effect:
```
kubectl delete pod -l app=<gateway-name>
```

SASL/OAUTHBEARER passthrough fails after a failover

Symptoms

Clients that use SASL/OAUTHBEARER identity passthrough to Confluent Cloud fail to authenticate after a failover.

Likely causes

Clients that use SASL/OAUTHBEARER identity passthrough send a logical cluster (lkc) extension to identify the target cluster. After a failover, the active cluster changes, so the client-supplied value becomes invalid and authentication fails.

Resolution

Set extensionHeaders in the route’s security.cluster entry for the streaming domain so that Confluent Gateway injects the correct logical cluster extension on behalf of the clients. This keeps client configurations independent of the active cluster.

For more information, see Configure identity passthrough.

Clients fail or stall on a fenced or migrated topic

Symptoms

Producers time out, or report NOT_ENOUGH_REPLICAS errors, for a topic.
Clients report NOT_LEADER_OR_FOLLOWER errors for a topic that you moved to another cluster.

Likely causes

A fencing rule still blocks the topic. Producers stop retrying when their delivery.timeout.ms expires.
Clients still use metadata from before you rerouted the topic. This is expected for a short time. Clients refresh their metadata and retry.

Resolution

Check for an active fence: A rising gateway_fencing_v2_requests_total rate for the topic means a fencing rule blocks it. Remove the rule from rules.fencing after you update rules.routing, and re-apply the CR:
```
kubectl apply -f <gateway-CR-file>
```
If spec.hotReload.enabled is false, Confluent Gateway pods roll to apply the change. For more information, see Configure routing rules (Early Access).
Verify the routing: Clients that receive NOT_LEADER_OR_FOLLOWER refresh their metadata and retry, so the errors stop shortly after a reroute. If they continue, verify that a rules.routing condition routes the topic to the cluster that owns it.

For the migration procedure, see Migrate Topics with Topic-Based Routing and Fencing using Confluent for Kubernetes. For more information about the metrics, see Monitor Confluent Private Cloud Gateway Metrics.

Requests on a dynamic route fail when a bound cluster is unavailable (Early Access)

Symptoms

Clients can’t connect, or Confluent Gateway closes their connections, on a route with streamingDomains.
Requests for topics on healthy clusters fail.

Likely causes

Confluent Gateway sends some requests, such as metadata and authentication requests, to every bound cluster, and the request fails if any of them is unavailable. This includes a retired cluster that you haven’t unbound from the route.

Resolution

Restore the cluster, or remove it from the route’s streamingDomains list and from any conditions, coordination, or default setting that references it. Then re-apply the CR. To check which requests fail, use the gateway_router_fanout_leg_failures_total metric.

For more information, see Configure routes.

License and route limit issues

Symptoms

The Confluent Gateway fails to initialize additional routes.
Logs report warnings on license or route-limit exhaustion.

Likely causes

Trial mode limits Confluent Gateway to a maximum of four routes. To configure more routes, you need an Enterprise license. For Confluent Gateway add-on for Confluent Platform, this is a Confluent Platform Enterprise license that includes the Confluent Gateway add-on.

Resolution

Apply an Enterprise license: Trial mode allows a maximum of four routes, and Enterprise mode supports unlimited routes. To create an Enterprise license secret, reference it from the CR, and review the license modes, see Configure Confluent Gateway license.
Inspect the license status: after re-applying the CR, confirm that the Confluent Gateway recognizes the license.
```
kubectl logs -l app=<gateway-name> | grep -i license
```