<a id="gateway-docker-troubleshoot"></a>

# Troubleshoot Confluent Gateway

Use this section to collect diagnostics and resolve common issues when running
Confluent Gateway with Docker.

## Capture Confluent Gateway logs manually

Use Docker commands to inspect the Confluent Gateway container logs.

```bash
# View all Confluent Gateway logs
docker logs gateway

# Filter for errors
docker logs gateway | grep -i error

# Filter for license-related messages
docker logs gateway | grep -i license
```

Begin troubleshooting by inspecting the Confluent Gateway container logs.
Most initialization failures, including network timeouts and credential
errors, are captured here.

## Enable debug logging to view stack traces

Set the global Confluent Gateway log level to `DEBUG` to capture detailed logs,
including stack traces. This helps diagnose complex issues when running
Confluent Gateway.

Update your `docker-compose.yaml` file to include the
`GATEWAY_ROOT_LOG_LEVEL` environment variable:

```yaml
gateway:
  environment:
    GATEWAY_ROOT_LOG_LEVEL: "DEBUG"
```

The `GATEWAY_ROOT_LOG_LEVEL` environment variable controls the global logging
level for Confluent Gateway, and the `DEBUG` log level captures detailed system
events, including stack traces.

## Known and common issues

### Confluent Gateway container exits or restarts repeatedly

**Symptoms**

* The Confluent Gateway container terminates after startup.
* Running the `docker ps` command shows the container is repeatedly
  restarting.
* Running the `docker compose up` command shows that the service
  failed to start.
* Logs contain configuration parsing errors, license validation
  failures, or stack traces.

**Likely causes**

* Invalid or incomplete Confluent Gateway configuration file.
* Misconfigured environment variables in the `docker-compose.yaml` file.

**Resolution**

1. Inspect the logs for configuration and license errors:
   ```bash
   docker logs gateway
   docker logs gateway | grep -i license
   ```

   Identify messages indicating failed YAML parsing, unknown fields, or
   unrecognized license keys.
2. Verify configuration file mapping. Confirm that the `volumes` section in
   your `docker-compose.yaml` file correctly maps the local configuration
   file to the expected path in the container.
3. Verify the configuration file structure:
   * Ensure that the `streamingDomains` section defines at least one
     streaming domain and a corresponding bootstrap server.
   * Ensure that the `routes` section defines at least one valid
     route definition.
4. Verify enterprise mode configuration:

   If you are using the enterprise license, confirm that the
   `GATEWAY_LICENSES` environment variable contains valid and
   unexpired license keys.

### Clients can’t connect to the Confluent Gateway endpoint

**Symptoms**

* Client connections to the Confluent Gateway endpoint either time out or
  stall indefinitely during metadata requests.
* Clients can directly connect to the Kafka cluster, but connections
  fail when routed through the Confluent Gateway.

**Likely causes**

* The Confluent Gateway container can’t reach the Kafka bootstrap servers due to
  DNS, firewall, or network issues.
* Clients are incorrectly configured to connect to broker addresses
  directly instead of the Confluent Gateway route endpoint.
* The `routes.endpoint` value doesn’t match the host and port
  published by the Docker container.
* The `bootstrapServerId` is mismatched across different sections of
  the configuration.

**Resolution**

1. Verify the bootstrap server configuration:

   Ensure the client’s `bootstrap.servers` property points to the
   route endpoint such as `gateway.example.com:19092`, rather than
   the broker addresses.
2. Align bootstrap IDs:

   Ensure that `bootstrapServerId` defined in the `streamingDomains`
   section matches the ID referenced in the `routes` section.
3. Confirm Docker port mappings:

   If the route endpoint is defined as `gateway.example.com:19092`,
   ensure that your `docker-compose.yaml` file publishes to that
   port. For example:
   ```yaml
   gateway:
     ports:
       - "19092:19092"
   ```
4. Test Confluent Gateway to Kafka connectivity:

   From within the container network or host where Confluent Gateway runs, test
   connectivity to the Kafka bootstrap endpoints defined under
   `streamingDomains.kafkaCluster.bootstrapServers[].endpoint`.

   Inspect DNS resolution, firewall rules, and VPC or VNet routing.
5. Review logs for authentication issues:

   Inspect logs for repeated connection attempts or `terminated during
   authentication` messages. If these errors occur, verify your
   authentication and SSL/TLS configuration settings.

### Protocol or SSL/TLS mismatch

**Symptoms**

* Client logs or Confluent Gateway logs show errors such as frame size
  exceptions or TLS handshake failures.
* Clients use `SASL_SSL` or `SSL` but the route is configured as
  plaintext, or vice versa.

**Likely causes**

* The client’s `security.protocol` doesn’t match the configured
  security for the route.
* TLS-encrypted traffic is directed to a plaintext route endpoint,
  or plaintext traffic is directed to an SSL-only route.

**Resolution**

1. Confirm route configuration:
   * Inspect the `gateway.routes[].security.ssl` settings in the
     Confluent Gateway configuration file.
   * If `security.ssl` is omitted, the route expects plaintext traffic.
   * If `security.ssl` is defined, configure the client to use an SSL
     protocol, such as, `SASL_SSL` or `SSL`.
2. Align client configuration:
   * For unencrypted connections, set the client protocol to `PLAINTEXT`.
   * When using SASL authentication without TLS, set the client
     protocol to `SASL_PLAINTEXT`.
   * For TLS routing, set the client protocol to `SSL` or `SASL_SSL`
     and configure the truststore and keystore as required.
3. After updating the configuration, restart the Confluent Gateway service to
   apply configuration changes and verify that client connectivity is
   restored.

### Authentication swap not behaving as expected

**Symptoms**

* Clients using authentication swap fail to authenticate with the
  backend cluster.
* Logs report errors related to secret store lookups, JAAS
  configuration failures, or callback handler initialization.
* Backend brokers report invalid credentials despite the client
  providing valid local certificates.

**Likely causes**

* Route security isn’t configured with `auth: swap`.
* The `secretStore` name referenced in the Confluent Gateway configuration is
  incorrect or missing in the `routes` section.
* Secret store provider type is incorrect, or the credentials required
  to access the store are misconfigured.

**Resolution**

1. Verify route swap configuration:

   Ensure the defined routes perform authentication swapping.
   ```yaml
   gateway:
     routes:
       - name: <route-name>
         security:
           auth: swap
           swapConfig: <swap-config>
   ```
2. Avoid hardcoded broker credentials for cluster authentication:

   Don’t embed static broker credentials in the Confluent Gateway configuration.
   For swap scenarios, credentials must be dynamically resolved from a
   supported provider, such as Amazon Secrets Manager, HashiCorp
   Vault, or a local file for dynamic resolution.
3. Verify your secret store configuration:
   * Provider types are case-sensitive. Verify that you are using the
     exact casing required. For example, use `Azure`, not `AZURE`.
   * Confirm the Confluent Gateway has the necessary permissions to read
     secrets from the provider.
   * Ensure the Confluent Gateway can reach the secret store’s API endpoint
     over the network.

<a id="gateway-passthrough-failover-troubleshoot"></a>

### Passthrough clients fail authentication after a failover

**Symptoms**

* Clients that use SASL/OAUTHBEARER identity passthrough authenticate
  successfully before a failover, but fail to authenticate with the backing
  Kafka cluster afterward.
* Logs report authentication failures that reference an invalid or unknown
  logical cluster (`lkc`) extension.

**Likely causes**

Clients that use SASL/OAUTHBEARER identity passthrough send a logical cluster
(`lkc`) extension to identify the target cluster. After a failover, the
active cluster changes, so the client-supplied value becomes invalid and
authentication fails.

**Resolution**

1. Configure `passthroughConfig` on the route so that Confluent Gateway injects the
   correct logical cluster extension on behalf of clients. This keeps client
   configurations independent of the active cluster.
   ```yaml
   gateway:
     routes:
       - name: <route_name>
         security:
           auth: passthrough
           passthroughConfig:
             sasl:
               mechanism: OAUTHBEARER
               extensionHeaders:
                 logicalCluster: <target-cluster-id>
   ```

   For configuration details, see [Configure identity passthrough mode](gateway-security.md#gateway-authn-passthrough-docker).
2. Restart the Confluent Gateway service to apply the configuration changes, then
   verify that clients reconnect and authenticate successfully.

<a id="gateway-oauth-endpoint-not-allowed"></a>

### OAuth token or JWKS endpoint not allowed

**Symptoms**

The Confluent Gateway fails to fetch OAuth tokens or validate JWTs against the
identity provider’s JWKS endpoint, and logs report a security
restriction similar to:
`... is not allowed. Update system property 'org.apache.kafka.sasl.oauthbearer.allowed.urls'
to allow https://...`

For client-side OAuth, an un-allowlisted JWKS endpoint (`jwksEndpointUrl`)
can also prevent the Confluent Gateway from starting, with an error such as
`The URL cannot be accessed due to restrictions`.

**Likely causes**

For security purposes, the Confluent Gateway requires an allowlist of outbound
OAuth URLs. The configured OAuth token endpoint or JWKS endpoint used by
the identity provider isn’t included in the default allowlist.

**Resolution**

1. Add the OAuth token endpoint, JWKS endpoint, or both to the
   `GATEWAY_OPTS` environment variable in the `docker-compose.yaml`
   file:
   ```yaml
   gateway:
     environment:
       GATEWAY_OPTS:
         -Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=
         https://login.microsoftonline.com/<tenantId>/oauth2/v2.0/token
   ```
2. Restart the Confluent Gateway service to apply the configuration changes.
   Check the logs to ensure the *not allowed* error no longer appears
   during the authentication handshake.

### License and route limit issues

**Symptoms**

* The Confluent Gateway fails to initialize additional routes.
* Logs report warnings on license or route-limit exhaustion.

**Likely causes**

The Confluent Gateway is running in trial mode, which has a maximum limit of
four routes.
To configure more routes, an enterprise license is required.

**Resolution**

1. Understand license modes:
   * **Trial mode (Default)**: Requires no license key. It is strictly
     limited to four routes.
   * **Enterprise mode**: Requires a valid Confluent Enterprise license for the
     Customer-Managed Confluent Platform for Confluent Cloud subscription. For license scope
     and coverage, see [Confluent Platform licenses](https://docs.confluent.io/platform/current/installation/license.html#enterprise-license-for-cmcpcc-subscription).
2. Apply an enterprise license: To enable more than four routes,
   add your license key to the `GATEWAY_LICENSES` environment variable
   in your `docker-compose.yaml` file.
3. Inspect the license status: After restarting the service,
   verify that the Confluent Gateway has successfully recognized the license.
   Run `docker logs gateway | grep -i license` to confirm the license status.