<a id="gateway-troubleshoot-docker"></a>

# Troubleshoot Confluent Gateway Docker Deployments

Use this page to collect diagnostics and resolve common issues when running
Confluent Gateway with Docker. Confluent Gateway deployment on Docker is officially supported
for production workloads. Start with the log-capture and debug-logging steps
that follow, then consult [Known and common issues](#gateway-known-common-issues) for
symptom-based resolutions.

## Capture Confluent Gateway logs manually

The Confluent Gateway container logs capture most initialization failures,
including network timeouts and credential errors. Inspect them first
using the following Docker commands:

```bash
# View all Gateway logs
docker logs gateway

# Filter for errors
docker logs gateway | grep -i error

# Filter for license-related messages
docker logs gateway | grep -i license
```

## Enable debug logging to view stack traces

The `DEBUG` log level captures detailed logs, including stack traces, to
help diagnose complex Confluent Gateway issues.

The `GATEWAY_ROOT_LOG_LEVEL` environment variable controls the global
logging level for Confluent Gateway. Update your `docker-compose.yaml` file to set
it to `DEBUG`:

```yaml
gateway:
  environment:
    GATEWAY_ROOT_LOG_LEVEL: "DEBUG"
```

<a id="gateway-known-common-issues"></a>

## Known and common issues

### Confluent Gateway container exits or restarts repeatedly

**Symptoms**

* The Confluent Gateway container terminates after startup.
* Running the `docker ps` command shows the container is repeatedly
  restarting.
* Running the `docker compose up` command shows that the service failed
  to start.
* Logs contain configuration parsing errors or stack traces.
* Logs report an unrecognized configuration field, such as
  `UnrecognizedPropertyException: Unrecognized field "tls"`, which
  indicates a misspelled or misplaced key.

**Likely causes**

* Invalid or incomplete Confluent Gateway configuration file.
* Misconfigured environment variables in the `docker-compose.yaml` file.

**Resolution**

1. **Inspect the logs for configuration and license errors**: Identify messages
   indicating failed YAML parsing or unknown fields.
   ```bash
   docker logs gateway
   docker logs gateway | grep -i license
   ```
2. **Verify the configuration source**: Confluent Gateway reads its configuration from
   one of the following, in order of precedence:
   * the `GATEWAY_CONFIG_FILE` environment variable (a configuration file)
   * the `GATEWAY_CONFIG` environment variable (inline YAML)
   * the `GATEWAY_CONFIG_TEMPLATE` file

   If you mount a configuration file, confirm that the `volumes` section in
   your `docker-compose.yaml` file maps it to the expected path in the container.
3. **Validate the configuration file structure**:
   * Ensure that the `streamingDomains` section defines at least one
     streaming domain and a corresponding bootstrap server.
   * Ensure that the `routes` section defines at least one valid route
     definition.
4. **Validate Enterprise mode configuration**: If you are using the Enterprise
   license, confirm that the `GATEWAY_LICENSES` environment variable
   contains valid and unexpired license keys.

### Clients can’t connect to the Confluent Gateway endpoint

**Symptoms**

* Client connections to the Confluent Gateway endpoint either time out or
  stall indefinitely during metadata requests.
* Clients can directly connect to the Kafka cluster, but connections fail
  when routed through the Confluent Gateway.

**Likely causes**

* The Confluent Gateway container can’t reach the Kafka bootstrap servers due to
  DNS, firewall, or network issues.
* Clients are incorrectly configured to connect to broker addresses directly
  instead of the Confluent Gateway route endpoint.
* The `routes.endpoint` value doesn’t match the host and port published by
  the Docker container.
* The `bootstrapServerId` is mismatched across different sections of the
  configuration.

**Resolution**

1. **Confirm Docker port mappings**: If the Confluent Gateway route endpoint is
   defined as `gateway.example.com:19092`, ensure that your
   `docker-compose.yaml` file publishes to that port. For example:
   ```yaml
   gateway:
     ports:
       - "19092:19092"
   ```
2. **Test Gateway to Kafka connectivity**:
   - From within the container network or host where Confluent Gateway runs, test
     connectivity to the Kafka bootstrap endpoints defined in
     `streamingDomains.kafkaCluster.bootstrapServers[].endpoint`.
   - Inspect DNS resolution, firewall rules, and network routing.
3. **Review logs for authentication issues**: Inspect logs for repeated
   connection attempts or `terminated during authentication` messages.
   If these errors occur, verify your authentication and SSL/TLS
   configuration settings. For more information on security configuration, see
   [Configure Security for Confluent Gateway](gateway-security.md#gateway-security-docker).
4. **Verify broker endpoint resolution**: When using the host-based
   broker identification strategy
   (`routes.brokerIdentificationStrategy.type: host`), verify that
   clients resolve individual broker addresses to the Confluent Gateway’s network
   load balancer (NLB). If your broker address pattern is defined as
   `b$(nodeId).gateway.example.com`, ensure that your DNS provider is
   configured to route these subdomains correctly.

   **Required DNS configuration**:

   To support dynamic broker identification, you must configure a wildcard
   DNS record pointing to your Confluent Gateway NLB:

   | Record type   | Host or pattern         | Target               |
   |---------------|-------------------------|----------------------|
   | A             | `gateway.example.com`   | `<gateway-nlb-host>` |
   | A (Wildcard)  | `*.gateway.example.com` | `<gateway-nlb-host>` |

   **Common misconfigurations to avoid**:

   Avoid the following settings, which prevent clients from connecting to
   the brokers:
   - **Invalid endpoint binding**: `route.endpoint = 0.0.0.0`

     **Result**: The Confluent Gateway advertises broker endpoints as `localhost`
     or `0.0.0.0` to the client. The client attempts to connect to its
     own local machine instead of the Confluent Gateway.
   - **Unresolvable host patterns**:
     `routes.brokerIdentificationStrategy.pattern` set to
     `b$(nodeId).<unresolvable-host>`

     **Result**: While the Confluent Gateway might start successfully, clients
     encounter `UnknownHostException` because the generated broker
     addresses don’t exist in your DNS registry.
   - **Missing wildcard records**: If you only map the root domain,
     such as `gateway.example.com`, without the wildcard (`*`), individual
     broker lookups like `b0.gateway.example.com` fail.

### Protocol or SSL/TLS mismatch

**Symptoms**

* Client logs or Confluent Gateway logs show errors such as frame size exceptions
  or TLS handshake failures.
* Clients use `SASL_SSL` or `SSL` but the route is configured as
  plaintext, or vice versa.

**Likely causes**

* The client’s `security.protocol` doesn’t match the configured security
  for the route.
* TLS-encrypted traffic is directed to a plaintext route endpoint,
  or plaintext traffic is directed to an SSL-only route.

**Resolution**

1. Confirm route configuration and restart the Confluent Gateway service to verify
   that client connectivity is restored.
   * Inspect the `gateway.routes[].security.ssl` settings in the
     Confluent Gateway configuration file.
   * If `security.ssl` is omitted, the route expects plaintext traffic.
   * If `security.ssl` is defined, configure the client to use an SSL
     protocol, such as `SASL_SSL` or `SSL`.
2. Align client configuration:
   * For unencrypted connections, set the client protocol to `PLAINTEXT`.
   * When using SASL authentication without TLS, set the client protocol
     to `SASL_PLAINTEXT`.
   * For TLS routing, set the client protocol to `SSL` or `SASL_SSL`
     and configure the truststore and keystore as required.

### Authentication swap not behaving as expected

**Symptoms**

* Clients using authentication swap fail to authenticate with the backend
  cluster.
* Logs report errors related to secret store lookups, Java Authentication
  and Authorization Service (JAAS) configuration failures, or callback
  handler initialization.
* Backend brokers report invalid credentials despite the client providing
  valid local certificates.

**Likely causes**

* The `clusterAuth` entry for the streaming domain isn’t configured with
  `auth: swap`. A streaming domain with no entry uses identity passthrough.
* The `secretStore` name referenced in the `clusterAuth` entry is incorrect
  or missing from `gateway.secretStores`.
* Secret store provider type is incorrect, or the credentials required to
  access the store are misconfigured.

**Resolution**

1. **Verify route swap configuration**: Ensure the defined routes perform
   authentication swapping.
   ```yaml
   gateway:
     routes:
       - name: <route_name>
         security:
           clientAuth: <client-auth>
           clusterAuth:
             <streaming-domain-name>:
               auth: swap
               secretStore: <secret-store-name>
               sasl: <cluster-sasl-settings>
   ```

   For the route authentication settings (Early Access), see
   [Route authentication configuration](gateway-security.md#gateway-authn-dynamic-docker). If your route uses
   `swapConfig`, see
   [Move a route to clientAuth and clusterAuth](gateway-security.md#gateway-authn-migrate-dynamic-docker).
2. **Avoid hard-coded broker credentials for cluster authentication**: Don’t
   embed static broker credentials in the Confluent Gateway configuration. For swap
   scenarios, credentials must be dynamically resolved from a supported
   provider such as HashiCorp Vault, a local file, or other supported
   secret stores for dynamic resolution.
3. **Validate your secret store configuration**:
   * Provider types are case-sensitive. Verify that you are using the exact
     casing required. For example, use `Azure`, not `AZURE`.
   * Confirm the Confluent Gateway has the necessary permissions to read secrets
     from the provider.
   * Ensure the Confluent Gateway can reach the secret store’s API endpoint over
     the network.

   For secret store configuration details, see
   [Configure Security for Confluent Gateway](gateway-security.md#gateway-security-docker).

### Passthrough clients fail authentication after a failover

**Symptoms**

* Clients that use SASL/OAUTHBEARER identity passthrough authenticate
  successfully before a failover, but fail to authenticate with the backing
  Kafka cluster afterward.
* Logs report authentication failures that reference an invalid or unknown
  logical cluster (`lkc`) extension.

**Likely causes**

Clients that use SASL/OAUTHBEARER identity passthrough send a logical cluster
(`lkc`) extension to identify the target cluster. After a failover, the
active cluster changes, so the client-supplied value becomes invalid and
authentication fails.

**Resolution**

1. Configure `passthroughConfig` on the route so that Confluent Gateway injects the
   correct logical cluster extension on behalf of clients. This keeps client
   configurations independent of the active cluster.
   ```yaml
   gateway:
     routes:
       - name: <route_name>
         security:
           auth: passthrough
           passthroughConfig:
             sasl:
               mechanism: OAUTHBEARER
               extensionHeaders:
                 logicalCluster: <target-cluster-id>
   ```

   For configuration details, see [Configure identity passthrough mode](gateway-security.md#gateway-authn-passthrough-docker).
2. If the route uses `clientAuth` and `clusterAuth` (Early Access),
   `passthroughConfig` isn’t valid on the route. Set the headers in the
   `clusterAuth` entry for the streaming domain instead:
   ```yaml
   gateway:
     routes:
       - name: <route_name>
         security:
           clusterAuth:
             <streaming-domain-name>:
               auth: passthrough
               sasl:
                 mechanism: OAUTHBEARER
                 extensionHeaders:
                   logicalCluster: <target-cluster-id>
   ```

   For all `clientAuth` and `clusterAuth` settings, see
   [Route authentication configuration](gateway-security.md#gateway-authn-dynamic-docker).
3. Restart the Confluent Gateway service to apply the configuration changes, then
   verify that clients reconnect and authenticate successfully.

### Clients fail or stall on a fenced or migrated topic

**Symptoms**

* Producers time out, or report `NOT_ENOUGH_REPLICAS` errors, for a topic.
* Consumers stop receiving messages for a topic, or can’t commit offsets or join
  their consumer group.
* Producers and consumers report `NOT_LEADER_OR_FOLLOWER` errors for a topic
  that you moved to another cluster.

**Likely causes**

* A fencing rule still blocks the topic. Producers stop retrying when their
  `delivery.timeout.ms` expires.
* Clients still use metadata from before you rerouted the topic. This
  behavior lasts only until clients refresh their metadata and retry.

**Resolution**

1. **Check for an active fence**: A rising
   `gateway_fencing_v2_requests_total` rate for the topic means a fencing rule
   blocks it. Remove the rule from `rules.fencing` after you update
   `rules.routing`, and apply the configuration.
2. **Verify the routing**: Clients that receive `NOT_LEADER_OR_FOLLOWER`
   refresh their metadata and retry, so the errors stop shortly after a
   reroute. If the errors continue, verify that a `rules.routing` condition
   routes the topic to the cluster that owns it.

For more information, see [Migrate topics between clusters](gateway-topic-based-routing.md#gateway-topic-based-routing-migrate) and
[Monitor topic-based routing and fencing](gateway-metrics-monitoring.md#gateway-metrics-topic-based-routing).

### Requests fail on a dynamic route with an unavailable cluster (Early Access)

**Symptoms**

* Clients can’t connect, or Confluent Gateway closes their connections, on a route
  with `streamingDomains`.
* Requests for topics on healthy clusters fail.

**Likely causes**

Confluent Gateway sends some requests, such as metadata and authentication requests, to
every bound cluster, and each of these requests fails if any bound cluster is
unavailable. A retired cluster that is still bound to the route also causes
these failures.

**Resolution**

Restore the cluster, or remove it from the route’s `streamingDomains` list
and from any `conditions`, `coordination`, or `default` setting that
references it. Then apply the configuration. To check which requests fail, use
`gateway_router_fanout_leg_failures_total`.

<a id="gateway-oauth-endpoint-not-allowed"></a>

### OAuth token endpoint not allowed

**Symptoms**

The Confluent Gateway fails to fetch OAuth tokens, and logs report a security
restriction similar to:
`... is not allowed. Update system property 'org.apache.kafka.sasl.oauthbearer.allowed.urls'
to allow https://...`

**Likely causes**

For security purposes, the Confluent Gateway requires an allowlist of outbound
OAuth requests. The configured OAuth token URL, or the JSON Web Key Set (JWKS)
endpoint URL that Confluent Gateway uses to validate client tokens, isn’t included in
the default allowlist.

**Resolution**

1. Add the OAuth token endpoint, and the JWKS endpoint if clients
   authenticate with `SASL/OAUTHBEARER`, to the `GATEWAY_OPTS`
   environment variable in the `docker-compose.yaml` file. Separate multiple
   URLs with commas:
   ```yaml
   gateway:
     environment:
       GATEWAY_OPTS: "-Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=<tokenEndpointUri>"
   ```

   Replace `<tokenEndpointUri>` with your identity provider’s OAuth
   token endpoint URL. For example, if you are using Microsoft Entra ID
   (formerly Azure AD), the token endpoint typically follows the pattern:
   `https://login.microsoftonline.com/<tenantId>/oauth2/v2.0/token`.
2. Restart the Confluent Gateway service to apply the configuration changes.
   Check the logs to ensure the **not allowed** error does not appear
   during the authentication handshake.

### License and route limit issues

**Symptoms**

* The Confluent Gateway fails to start or to apply a configuration with more than
  four routes.
* Logs report `Cannot configure <n> routes. License allows maximum 4
  routes.`

**Likely causes**

Trial mode limits Confluent Gateway to a maximum of four routes. To configure more
routes, a Confluent Enterprise license for Confluent Platform that includes the Confluent Gateway add-on is required.

**Resolution**

1. **Understand license modes**: For a comparison of Trial and Enterprise
   modes, see [License modes](gateway-deploy-overview.md#gateway-license-modes).
2. **Apply an Enterprise license**: To enable more than four routes,
   add your license key, which must include the Confluent Gateway add-on (see
   [Manage Confluent Platform Licenses](../installation/license.md#cp-license-overview)), to the `GATEWAY_LICENSES` environment variable
   in your `docker-compose.yaml` file. For configuration details and
   examples, see [Optional: Enterprise license configuration](gateway-deploy.md#gateway-config-license-docker).
3. **Inspect the license status**: After restarting the service,
   verify that the Confluent Gateway has successfully recognized the license.
   Run `docker compose -f docker-compose.yaml logs gateway | grep -i license`
   to confirm the license status.

### Container fails to start on a read-only file system

**Symptoms**

* The Confluent Gateway container fails to start, and logs show an error such as
  `Unable to create file /usr/logs/gateway.log` or
  `java.io.IOException: Read-only file system`.
* Logs report log4j `RollingFileAppender` initialization failures.

**Likely causes**

Confluent Gateway writes log files to `/usr/logs` and its rendered configuration
to `/etc/gateway`. Running with a read-only root file system, for example
with the `--read-only` flag or a read-only volume mount, blocks these
writes and prevents the container from starting.

**Resolution**

1. Confirm that the error references a path that Confluent Gateway must write to,
   such as the log directory at `/usr/logs` or the configuration directory
   at `/etc/gateway`.
2. Mount writable volumes for `/usr/logs` and `/etc/gateway`. If you run
   the container with `--read-only`, add a writable `tmpfs` or volume
   mount for these paths.