<a id="co-gateway-troubleshoot"></a>

# Troubleshoot Confluent Gateway CFK Deployments

Diagnose and resolve common Confluent Gateway issues
on Confluent for Kubernetes (CFK). Start with the log-capture and debug-logging
steps that follow, then consult [Known and common issues](#co-gateway-known-common-issues) for
symptom-based resolutions.

#### NOTE
The Kubernetes commands on this page assume that you have already set the
default namespace. If Confluent Gateway runs outside the default namespace, add
`-n <namespace>` to each command.

## Capture Confluent Gateway logs manually

The Confluent Gateway pod logs capture most initialization failures, including network
timeouts and credential errors. Inspect them first using the following
Kubernetes commands:

```bash
# Get the Gateway custom resource
kubectl get Gateway <gateway_name> -o yaml > gateway.yaml

# List Gateway pods
kubectl get pods -l app=confluent-gateway

# View logs from all Gateway pods
kubectl logs -l app=confluent-gateway

# Filter for errors
kubectl logs -l app=confluent-gateway | grep -i error
```

Review these logs first to identify configuration or connectivity issues before
you investigate Confluent Gateway routing or client settings.

## Enable debug logging to view stack traces

The `DEBUG` log level captures detailed logs, including stack traces, to help
diagnose complex Confluent Gateway issues.

The `GATEWAY_ROOT_LOG_LEVEL` environment variable controls the global logging
level for Confluent Gateway. Update the `podTemplate` in your custom resource (CR) to
set it to `DEBUG`, then re-apply the CR:

```yaml
kind: Gateway
spec:
  podTemplate:
    envVars:
      - name: GATEWAY_ROOT_LOG_LEVEL
        value: DEBUG
```

## Capture Confluent Gateway logs with a support bundle

Use the CFK support bundle command to collect only Confluent Gateway-related
diagnostics, including logs, CRs, and related resources:

```bash
kubectl confluent support-bundle --only-gateway
```

Use the generated archive when sharing details with Confluent Support.

## Verify client-to-Kafka connectivity

Test a client that connects directly to the backend Apache Kafka® cluster, bypassing
Confluent Gateway, to rule out baseline cluster or network issues before
troubleshooting Confluent Gateway routing.

<a id="co-gateway-known-common-issues"></a>

## Known and common issues

### Confluent Gateway pods in CrashLoopBackOff

**Symptoms**

* The Confluent Gateway pods restart repeatedly or remain in `CrashLoopBackOff`.
* Running the `kubectl get pods -l app=confluent-gateway` command shows
  repeated restarts.
* Pod logs contain configuration parsing errors, license validation failures,
  or stack traces.
* Logs report an unrecognized configuration field, such as
  `UnrecognizedPropertyException: Unrecognized field "tls"`, which indicates
  a misspelled or misplaced key.

**Likely causes**

* Invalid or incomplete Confluent Gateway CR, such as malformed `streamingDomains`,
  `routes`, or secret references.
* A missing or invalid license secret that `GATEWAY_LICENSES` references.

**Resolution**

1. Inspect the pod logs for configuration and license errors:
   ```bash
   kubectl logs -l app=confluent-gateway
   kubectl logs -l app=confluent-gateway | grep -i license
   ```
2. Validate the CR structure:
   * Ensure that the `streamingDomains` section defines at least one
     streaming domain and a corresponding bootstrap server.
   * Ensure that the `routes` section defines at least one valid route.
3. Verify secret references: confirm that any `secretStores` and license
   secrets referenced in the CR exist in the same namespace and contain the
   expected keys.
4. Correct the configuration and re-apply the CR:
   ```bash
   kubectl apply -f <gateway-CR-file>
   ```

### Clients can’t connect to the Confluent Gateway endpoint

**Symptoms**

* Client connections to the Confluent Gateway endpoint time out or stall indefinitely
  during metadata requests.
* Clients connect successfully when they target the Kafka cluster directly, but
  fail to connect when routed through Confluent Gateway.

**Likely causes**

* The Confluent Gateway pods can’t reach the Kafka bootstrap servers due to DNS,
  firewall, or network policy issues.
* Clients connect to broker addresses directly instead of the Confluent Gateway route
  endpoint.
* The `routes.endpoint` value doesn’t match the address published by the
  Confluent Gateway Kubernetes service.
* The `bootstrapServerId` does not match across sections of the CR.

**Resolution**

1. Verify that the route endpoint is reachable: confirm that the Confluent Gateway
   Kubernetes service or load balancer exposes the host and port defined in
   `routes.endpoint`.
   ```bash
   kubectl get svc -l app=confluent-gateway
   ```
2. Test Confluent Gateway-to-Kafka connectivity: from a Confluent Gateway pod, test
   connectivity to the Kafka bootstrap endpoints defined in
   `streamingDomains.kafkaCluster.bootstrapServers[].endpoint`. Inspect DNS
   resolution, firewall rules, and network policies.
3. Review logs for authentication issues: inspect logs for repeated connection
   attempts or `terminated during authentication` messages. If these errors
   occur, verify your authentication and TLS/SSL configuration settings. For
   more information, see [Configure Security for Confluent Gateway using Confluent for Kubernetes](co-gateway-security.md#co-gateway-security).
4. Verify broker endpoint resolution: when you use the host-based broker
   identification strategy, where
   `routes.brokerIdentificationStrategy.type` is `host`, clients must
   resolve individual broker addresses to the Confluent Gateway endpoint. If your
   broker address pattern is
   `broker-$(nodeId).gateway.example.com`, ensure that your DNS provider
   routes these subdomains, including a wildcard record, to the Confluent Gateway
   service.

### Protocol or TLS/SSL mismatch

**Symptoms**

* Client logs or Confluent Gateway logs show errors such as frame size exceptions (for
  example, `FrameOversizedException`) or TLS handshake failures.
* Clients use `SASL_SSL` or `SSL`, but the route uses plaintext, or vice
  versa.

**Likely causes**

* The client’s `security.protocol` doesn’t match the configured security for
  the route.
* TLS-encrypted traffic reaches a plaintext route endpoint, or plaintext
  traffic reaches an SSL-only route.

**Resolution**

1. Confirm the route configuration: inspect the
   `routes[].security.client.tls` settings in the Confluent Gateway CR.
   * If you omit `security.client.tls`, the route expects plaintext
     traffic.
   * If you define `security.client.tls`, configure the client to use an
     SSL protocol, such as `SASL_SSL` or `SSL`.
2. Align the client configuration:
   * For unencrypted connections, set the client protocol to `PLAINTEXT`.
   * When using SASL authentication without TLS, set the client protocol to
     `SASL_PLAINTEXT`.
   * For TLS routing, set the client protocol to `SSL` or `SASL_SSL` and
     configure the truststore and keystore as required.
3. Re-apply the CR if you change route security, and confirm that clients can
   connect again.

### Authentication swap not behaving as expected

**Symptoms**

* Clients using authentication swap fail to authenticate with the backend
  cluster.
* Logs report errors related to secret store lookups, Java Authentication and
  Authorization Service (JAAS) configuration failures, or callback handler
  initialization.
* Backend brokers report invalid credentials despite the client providing valid
  local credentials.

**Likely causes**

* Route security isn’t configured with `auth: swap`.
* The `secretStore` referenced in the `routes` section is incorrect or
  missing.
* The secret store provider type is incorrect, or the credentials required to
  access the store are incorrect.

**Resolution**

1. Verify the route swap configuration: ensure that the route performs
   authentication swapping.
   ```yaml
   kind: Gateway
   spec:
     routes:
       - name: <route_name>
         security:
           auth: swap
   ```
2. Avoid hard-coded broker credentials: don’t embed static broker credentials
   in the CR. For swap scenarios, Confluent Gateway must resolve credentials
   dynamically from a configured secret store.
3. Validate your secret store configuration:
   * Provider types are case-sensitive. Verify that you use the exact casing
     required. For example, use `Azure`, not `AZURE`.
   * Confirm that the Confluent Gateway has the necessary permissions to read secrets
     from the provider and can reach the provider’s API endpoint over the
     network.

   To learn more about secret store configuration, see
   [Configure Security for Confluent Gateway using Confluent for Kubernetes](co-gateway-security.md#co-gateway-security).
4. Restart the Confluent Gateway pods so the updated authentication settings take
   effect:
   ```bash
   kubectl delete pod -l app=confluent-gateway
   ```

### SASL/OAUTHBEARER passthrough fails after a failover

**Symptoms**

Clients that use SASL/OAUTHBEARER identity passthrough to Confluent Cloud fail to
authenticate after a failover.

**Likely causes**

Clients that use SASL/OAUTHBEARER identity passthrough send a logical cluster
(`lkc`) extension to identify the target cluster. After a failover, the
active cluster changes, so the client-supplied value becomes invalid and
authentication fails.

**Resolution**

Configure `passthroughConfig` on the route so that Confluent Gateway injects the
correct logical cluster extension on behalf of the clients. This keeps client
configurations independent of the active cluster.

For more information, see [Configure identity passthrough](co-gateway-security.md#co-gateway-authn-passthrough).

### License and route limit issues

**Symptoms**

* The Confluent Gateway fails to initialize additional routes.
* Logs report warnings on license or route-limit exhaustion.

**Likely causes**

Trial mode limits Confluent Gateway to a maximum of four routes. To configure more
routes, you need an Enterprise license.

**Resolution**

1. Apply an Enterprise license: Trial mode allows a maximum of four routes, and
   Enterprise mode supports unlimited routes. To create an Enterprise license
   secret, reference it from the CR, and review the license modes, see
   [Configure Confluent Gateway license](co-gateway-deploy.md#co-gateway-config-license).
2. Inspect the license status: after re-applying the CR, confirm that the
   Confluent Gateway recognizes the license.
   ```bash
   kubectl logs -l app=confluent-gateway | grep -i license
   ```