Monitor Confluent Gateway Metrics

Confluent Gateway exposes Prometheus-format metrics and a liveness check through a single administrative service. For information on how to enable the metrics endpoint, see Administration and metrics configuration.

Metric names and labels can change between Confluent Gateway releases. Query the administrative service’s /metrics endpoint at http://{bindAddress}:{port}/metrics directly on your Confluent Gateway instance to verify the exact metric names exposed by your deployed version.

To view pre-built Grafana dashboards and a sample Prometheus stack you can run locally, see the gateway monitoring examples in the confluent-kubernetes-examples repository.

Monitor failed connections

Use these metrics to detect downstream request failures and upstream broker connections that fail to establish.

Failed connection metrics

Metric	Description	Alert guidance
kroxylicious_client_to_proxy_errors_total	Total count of downstream client-facing errors. Measures failures occurring on the proxy before requests reach upstream Kafka brokers.	Alert on a sustained non-zero rate over five minutes.
kroxylicious_client_to_proxy_connections_total and kroxylicious_proxy_to_server_connections_total	Measures cumulative connection attempts from clients to the proxy with connection attempts from the proxy to upstream brokers. Divergence indicates the proxy accepts client connections but fails to establish broker connections.	Alert if the rates of these two metrics diverge for more than five minutes. Compare rates, for example with rate(...[5m]), because both metrics are cumulative counters.
kroxylicious_client_to_proxy_disconnects_total {cause}	Measures total client disconnections partitioned by cause, such as graceful shutdowns with client_closed, session timeouts with idle_timeout, or active node drains with drain_timeout.	Alert on spikes in drain_timeout or any cause other than client_closed.
kroxylicious_virtual_cluster_state{state}	Identifies current lifecycle state of virtual cluster routes, returning 1 for the active state and 0 for inactive states. Valid states include serving, failed, draining, initializing, and stopped.	Alert if any route reports state="failed" with a value of 1.

Example Prometheus Query Language (PromQL) query for the downstream error rate:

rate(kroxylicious_client_to_proxy_errors_total[5m])

Track request and response completion

Use these metrics to identify requests that do not complete and schema validation failures on enforced routes.

Request and response completion metrics

Metric	Description	Alert guidance
kroxylicious_client_to_proxy_request_total compared to kroxylicious_proxy_to_client_response_total or kroxylicious_server_to_proxy_response_total	Requests that never receive a response.	Alert if the ratio of rate() values for responses to requests drifts from 1 for more than five minutes. These metrics are cumulative counters, so compare rates and not lifetime totals.
gateway_governance_data_latency_seconds_count {outcome}	Count of partitions that Confluent Gateway validated, per route, topic, and mode, with the outcome and failure origin. Confluent Gateway emits this metric only for routes that have governance enabled. For more information, see Enforce Data Governance Centrally with Confluent Gateway.	Alert on a sustained outcome="rejected" rate.

Example PromQL queries:

# Request rate
rate(kroxylicious_client_to_proxy_request_total[5m])

# Response rate
rate(kroxylicious_proxy_to_client_response_total[5m])

# Governance rejection rate by route and topic
sum by (route, topic) (rate(gateway_governance_data_latency_seconds_count{outcome="rejected"}[5m]))

Measure throughput and latency

Use these metrics for capacity planning and to track authentication swapping (AuthSwap) latency against your service-level objectives (SLOs).

Throughput and latency metrics

Metric	Description	Alert guidance
kroxylicious_*_request_size_bytes_* and kroxylicious_*_response_size_bytes_*	Traffic volume and payload sizes for client-to-proxy and proxy-to-broker traffic.	Track for capacity planning. No default alert required.
gateway_authswap_latency_seconds{quantile}	Client-observed end-to-end AuthSwap latency percentiles.	Alert if the quantile="0.99" series exceeds your SLO.
gateway_authswap_secret_store_latency_seconds {quantile,type}	Secret store lookup latency during AuthSwap.	Alert if the quantile="0.99" series exceeds your SLO.
gateway_authswap_client_auth_total{result} and gateway_authswap_cluster_auth_total{result}	Client-side and cluster-side AuthSwap authentication success and failure counts.	Alert on a sustained result="failure" rate.

Note

Confluent Gateway emits gateway_authswap_* metrics only for routes configured with AuthSwap authentication. Routes using passthrough authentication do not emit these series. For details, see Authentication swapping.

Example PromQL queries:

# Average request size by virtual cluster
rate(kroxylicious_client_to_proxy_request_size_bytes_sum[5m])
  / rate(kroxylicious_client_to_proxy_request_size_bytes_count[5m])

# AuthSwap client authentication failure rate by route
rate(gateway_authswap_client_auth_total{result="failure"}[5m])

# 99th-percentile AuthSwap latency by route
gateway_authswap_latency_seconds{quantile="0.99"}

Monitor topic-based routing and fencing

Use these metrics to monitor fencing rules and dynamic routes. Dynamic routing is an Early Access feature. For more information, see Migrate Topics Incrementally with Topic-Based Routing and Fencing.

Topic-based routing and fencing metrics

Metric	Description	Alert guidance
gateway_fencing_v2_requests_total {plane,topic}	Requests that fencing rules blocked. plane: produce or consume. topic: The topic name, or UNKNOWN for requests that don’t name a topic, such as consumer group requests.	A non-zero rate indicates an active fence. Alert if the rate doesn’t return to zero after you remove a fence.
gateway_router_rejections_total {virtualCluster,apiKey,reason}	Requests that a dynamic route rejected. virtualCluster: The name of the Confluent Gateway route. apiKey: The Kafka API of the rejected request. reason: The rejection cause, which is one of unadvertised-api, bootstrap-connection, non-consumer-replica, unsupported-key-type, transactional-id, cross-cluster-ownership, or unresolvable-topic.	After you reroute a topic, expect a transient spike in cross-cluster-ownership. Alert on a sustained rate for any other reason.
gateway_router_fanout_leg_failures_total {virtualCluster,apiKey}	Fan-out requests that failed on a bound cluster. Confluent Gateway sends a fan-out request to every bound cluster, and a failure on any one cluster fails the request and closes the client connection. virtualCluster: The name of the Confluent Gateway route. apiKey: The Kafka API of the failed request.	Alert on any non-zero rate.
gateway_router_metadata_topics_dropped_total {virtualCluster,reason}	Topics that Confluent Gateway filtered out of metadata responses. virtualCluster: The name of the Confluent Gateway route. reason: Why Confluent Gateway dropped the topic. owned-elsewhere: A cluster that doesn’t own the topic returned it. This is normal on a route bound to more than one streaming domain. unowned: The route is strict, and the topic matches no condition and no default.	Not an alert signal. The rate is non-zero during normal operation on any route bound to more than one streaming domain. To calculate the average number of dropped topics per metadata request, divide by the total number of metadata calls (the next metric in this table).
gateway_router_metadata_calls_total {virtualCluster}	Metadata requests that Confluent Gateway handled on dynamic routes. virtualCluster: The name of the Confluent Gateway route.	Use as the denominator for the dropped-topics metric in the previous row.
gateway_router_producer_id_rewrites_total {virtualCluster}	Producer ID rewrites that Confluent Gateway performed on dynamic routes. virtualCluster: The name of the Confluent Gateway route.	No default alert required.

Example PromQL queries:

# Blocked requests by plane and topic
sum by (plane, topic) (rate(gateway_fencing_v2_requests_total[5m]))

# Rejected requests by reason
sum by (reason) (rate(gateway_router_rejections_total[5m]))

# Average number of dropped topics per metadata request
sum(rate(gateway_router_metadata_topics_dropped_total[5m]))
  / sum(rate(gateway_router_metadata_calls_total[5m]))

Check process health

Use standard Java Virtual Machine (JVM) and process metrics to monitor process health.

Process health metrics

Metric	Description	Alert guidance
jvm_memory_used_bytes and jvm_memory_max_bytes	Heap and non-heap memory usage compared to configured limits.	Alert if usage exceeds 85% of maximum memory.
jvm_gc_pause_seconds	Garbage collection pause time.	Alert on a rising rate(..._sum[5m]).
process_cpu_usage and system_cpu_usage	Process CPU usage compared to total host CPU usage.	Alert if usage exceeds your configured threshold.
process_files_open_files and process_files_max_files	Open file descriptors compared to the maximum limit. Requires FileDescriptorMetrics in the admin.jvmMetrics configuration. For more information, see Administration and metrics configuration.	Alert if open files exceed 80% of the maximum limit.
kroxylicious_client_to_proxy_active_connections and kroxylicious_proxy_to_server_active_connections	Active client-to-proxy and proxy-to-broker connection counts.	Track against expected client and broker counts.
netty_eventexecutor_tasks_pending	Pending tasks per Netty event-loop thread.	Alert on a sustained non-zero value to detect event-loop saturation.

Example PromQL query for heap usage:

sum(jvm_memory_used_bytes{area="heap"}) / sum(jvm_memory_max_bytes{area="heap"} > 0)