# Monitor Confluent Gateway Metrics Confluent Gateway exposes Prometheus-format metrics and a liveness check through a single administrative service. For information on how to enable the metrics endpoint, see [Administration and metrics configuration](gateway-deploy.md#gateway-config-admin-metrics-docker). Metric names and labels can change between Confluent Gateway releases. Query the administrative service’s `/metrics` endpoint at `http://{bindAddress}:{port}/metrics` directly on your Confluent Gateway instance to verify the exact metric names exposed by your deployed version. To view pre-built Grafana dashboards and a sample Prometheus stack you can run locally, see the [gateway monitoring examples](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/gateway/monitoring) in the `confluent-kubernetes-examples` repository. ## Monitor failed connections Use these metrics to detect downstream request failures and upstream broker connections that fail to establish. #### Failed connection metrics | Metric | Description | Alert guidance | |-----------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `kroxylicious_client_to_proxy_errors_total` | Total count of downstream client-facing errors. Measures failures
occurring on the proxy before requests reach upstream Kafka brokers. | Alert on a sustained non-zero rate over five minutes. | | `kroxylicious_client_to_proxy_connections_total` and
`kroxylicious_proxy_to_server_connections_total` | Measures cumulative connection attempts from clients to the proxy with
connection attempts from the proxy to upstream brokers. Divergence
indicates the proxy accepts client connections but fails to
establish broker connections. | Alert if the rates of these two metrics diverge for more than five
minutes. Compare rates, for example with `rate(...[5m])`, because
both metrics are cumulative counters. | | `kroxylicious_client_to_proxy_disconnects_total`
`{cause}` | Measures total client disconnections partitioned by cause, such as graceful
shutdowns with `client_closed`, session timeouts with `idle_timeout`,
or active node drains with `drain_timeout`. | Alert on spikes in `drain_timeout` or any cause other than
`client_closed`. | | `kroxylicious_virtual_cluster_state{state}` | Identifies current lifecycle state of virtual cluster routes, returning `1`
for the active state and `0` for inactive states. Valid states
include `serving`, `failed`, `draining`, `initializing`,
and `stopped`. | Alert if any route reports `state="failed"` with a value of `1`. | Example Prometheus Query Language (PromQL) query for the downstream error rate: ```promql rate(kroxylicious_client_to_proxy_errors_total[5m]) ``` ## Track request and response completion Use these metrics to identify requests that do not complete and schema validation failures on enforced routes. #### Request and response completion metrics | Metric | Description | Alert guidance | |-----------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `kroxylicious_client_to_proxy_request_total` compared to
`kroxylicious_proxy_to_client_response_total` or
`kroxylicious_server_to_proxy_response_total` | Requests that never receive a response. | Alert if the ratio of `rate()` values for responses to requests
drifts from `1` for more than five minutes. These metrics are
cumulative counters, so compare rates and not lifetime totals. | | `gateway_governance_data_latency_seconds_count`
`{outcome}` | Count of partitions that Confluent Gateway validated, per route, topic, and
mode, with the outcome and failure origin. Confluent Gateway emits this metric
only for routes that have governance enabled. For more information, see
[Enforce Data Governance Centrally with Confluent Gateway](governance/gateway-governance.md#gateway-governance). | Alert on a sustained `outcome="rejected"` rate. | Example PromQL queries: ```promql # Request rate rate(kroxylicious_client_to_proxy_request_total[5m]) # Response rate rate(kroxylicious_proxy_to_client_response_total[5m]) # Governance rejection rate by route and topic sum by (route, topic) (rate(gateway_governance_data_latency_seconds_count{outcome="rejected"}[5m])) ``` ## Measure throughput and latency Use these metrics for capacity planning and to track authentication swapping (AuthSwap) latency against your service-level objectives (SLOs). #### Throughput and latency metrics | Metric | Description | Alert guidance | |----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|---------------------------------------------------------| | `kroxylicious_*_request_size_bytes_*` and
`kroxylicious_*_response_size_bytes_*` | Traffic volume and payload sizes for client-to-proxy and
proxy-to-broker traffic. | Track for capacity planning. No default alert required. | | `gateway_authswap_latency_seconds{quantile}` | Client-observed end-to-end AuthSwap latency percentiles. | Alert if the `quantile="0.99"` series exceeds your SLO. | | `gateway_authswap_secret_store_latency_seconds`
`{quantile,type}` | Secret store lookup latency during AuthSwap. | Alert if the `quantile="0.99"` series exceeds your SLO. | | `gateway_authswap_client_auth_total{result}` and
`gateway_authswap_cluster_auth_total{result}` | Client-side and cluster-side AuthSwap authentication success and
failure counts. | Alert on a sustained `result="failure"` rate. | #### NOTE Confluent Gateway emits `gateway_authswap_*` metrics only for routes configured with AuthSwap authentication. Routes using passthrough authentication do not emit these series. For details, see [Authentication swapping](gateway-security.md#gateway-authn-swap-docker). Example PromQL queries: ```promql # Average request size by virtual cluster rate(kroxylicious_client_to_proxy_request_size_bytes_sum[5m]) / rate(kroxylicious_client_to_proxy_request_size_bytes_count[5m]) # AuthSwap client authentication failure rate by route rate(gateway_authswap_client_auth_total{result="failure"}[5m]) # 99th-percentile AuthSwap latency by route gateway_authswap_latency_seconds{quantile="0.99"} ``` ## Monitor topic-based routing and fencing Use these metrics to monitor fencing rules and dynamic routes. Dynamic routing is an Early Access feature. For more information, see [Migrate Topics Incrementally with Topic-Based Routing and Fencing](gateway-topic-based-routing.md#gateway-topic-based-routing). #### Topic-based routing and fencing metrics | Metric | Description | Alert guidance | |------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `gateway_fencing_v2_requests_total`
`{plane,topic}` | Requests that fencing rules blocked.

* `plane`: `produce` or `consume`.
* `topic`: The topic name, or `UNKNOWN` for requests that don’t
name a topic, such as consumer group requests. | A non-zero rate indicates an active fence. Alert if the rate doesn’t
return to zero after you remove a fence. | | `gateway_router_rejections_total`
`{virtualCluster,apiKey,reason}` | Requests that a dynamic route rejected.

* `virtualCluster`: The name of the Confluent Gateway route.
* `apiKey`: The Kafka API of the rejected request.
* `reason`: The rejection cause, which is one of
`unadvertised-api`, `bootstrap-connection`,
`non-consumer-replica`, `unsupported-key-type`,
`transactional-id`, `cross-cluster-ownership`, or
`unresolvable-topic`. | After you reroute a topic, expect a transient spike in
`cross-cluster-ownership`. Alert on a sustained rate for any other
reason. | | `gateway_router_fanout_leg_failures_total`
`{virtualCluster,apiKey}` | Fan-out requests that failed on a bound cluster. Confluent Gateway sends a
fan-out request to every bound cluster, and a failure on any one cluster
fails the request and closes the client connection.

* `virtualCluster`: The name of the Confluent Gateway route.
* `apiKey`: The Kafka API of the failed request. | Alert on any non-zero rate. | | `gateway_router_metadata_topics_dropped_total`
`{virtualCluster,reason}` | Topics that Confluent Gateway filtered out of metadata responses.

* `virtualCluster`: The name of the Confluent Gateway route.
* `reason`: Why Confluent Gateway dropped the topic.
* `owned-elsewhere`: A cluster that doesn’t own the topic returned
it. This is normal on a route bound to more than one streaming
domain.
* `unowned`: The route is strict, and the topic matches no
condition and no `default`. | Not an alert signal. The rate is non-zero during normal operation on any
route bound to more than one streaming domain. To calculate the
average number of dropped topics per metadata request, divide by the
total number of metadata calls (the next metric in this table). | | `gateway_router_metadata_calls_total`
`{virtualCluster}` | Metadata requests that Confluent Gateway handled on dynamic routes.

* `virtualCluster`: The name of the Confluent Gateway route. | Use as the denominator for the dropped-topics metric in the previous
row. | | `gateway_router_producer_id_rewrites_total`
`{virtualCluster}` | Producer ID rewrites that Confluent Gateway performed on dynamic routes.

* `virtualCluster`: The name of the Confluent Gateway route. | No default alert required. | Example PromQL queries: ```promql # Blocked requests by plane and topic sum by (plane, topic) (rate(gateway_fencing_v2_requests_total[5m])) # Rejected requests by reason sum by (reason) (rate(gateway_router_rejections_total[5m])) # Average number of dropped topics per metadata request sum(rate(gateway_router_metadata_topics_dropped_total[5m])) / sum(rate(gateway_router_metadata_calls_total[5m])) ``` ## Check process health Use standard Java Virtual Machine (JVM) and process metrics to monitor process health. #### Process health metrics | Metric | Description | Alert guidance | |-------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------| | `jvm_memory_used_bytes` and `jvm_memory_max_bytes` | Heap and non-heap memory usage compared to configured limits. | Alert if usage exceeds 85% of maximum memory. | | `jvm_gc_pause_seconds` | Garbage collection pause time. | Alert on a rising `rate(..._sum[5m])`. | | `process_cpu_usage` and `system_cpu_usage` | Process CPU usage compared to total host CPU usage. | Alert if usage exceeds your configured threshold. | | `process_files_open_files` and `process_files_max_files` | Open file descriptors compared to the maximum limit. Requires
`FileDescriptorMetrics` in the `admin.jvmMetrics` configuration.
For more information, see
[Administration and metrics configuration](gateway-deploy.md#gateway-config-admin-metrics-docker). | Alert if open files exceed 80% of the maximum limit. | | `kroxylicious_client_to_proxy_active_connections` and
`kroxylicious_proxy_to_server_active_connections` | Active client-to-proxy and proxy-to-broker connection counts. | Track against expected client and broker counts. | | `netty_eventexecutor_tasks_pending` | Pending tasks per Netty event-loop thread. | Alert on a sustained non-zero value to detect
event-loop saturation. | Example PromQL query for heap usage: ```promql sum(jvm_memory_used_bytes{area="heap"}) / sum(jvm_memory_max_bytes{area="heap"} > 0) ```