Confluent Cloud Metrics

The Confluent Cloud Metrics API is a REST API that returns time-series metrics for Apache Kafka® clusters, connectors, Stream Governance components, and Apache Flink® compute pools. Use it to query throughput, consumer lag, partition count, request rates, and other operational metrics. Metrics are also available through integrations with Datadog, Prometheus, Grafana, and other third-party monitoring platforms. For ready-to-use queries covering these resource types, see Run Example Queries for the Metrics API.

Metrics quick start

Use the Metrics quick start to start working with the metrics that Confluent Cloud provides. The Metrics API supports a diverse set of querying patterns that provide usage and performance analysis over time. To query throughput, consumer lag, and other operational metrics, use the Confluent Cloud Metrics API.

Considerations

• You must use an API key resource-scoped for resource management to communicate with the Metrics API.

• API keys resource-scoped for a Kafka cluster cause an authentication error.

• Metrics in Confluent Cloud are available either through first-class integrations with third-party monitoring providers or by directly querying the Confluent Cloud Metrics API.

• To reduce the operational burden of monitoring, use an integration with third-party monitoring providers.

Prerequisites

• API key resource-scoped for resource management.

• Confluent Cloud resources to monitor.

Get started: Create an API key and run example queries. For more information, see Create an API key to authenticate to the Metrics API and Run Example Queries for the Confluent Cloud Metrics API.

See also

For an example that showcases how to monitor a Kafka client application and Confluent Cloud metrics, and steps through various failure scenarios to show metrics results, see the Observability for Kafka Clients to Confluent Cloud.

Create an API key to authenticate to the Metrics API

To securely access Confluent Cloud metrics programmatically or through third-party monitoring tools, you need both the MetricsViewer role for authorization and an API key for authentication.

The MetricsViewer role provides service account access to the Metrics API for all clusters in an organization. This role also enables service accounts to import metrics into third-party metrics platforms.

To create an API key to authenticate to the Metrics API:

Confluent CLI

Run the following commands to add a role binding for MetricsViewer to a new service account. Remember to log in with the confluent login command first.

1. Create the service account:

   confluent iam service-account create MetricsImporter --description "A test service account to import Confluent Cloud metrics into our monitoring system"
   

   Your output should resemble:

   +-------------+--------------------------------+
   | ID          | sa-123abc                      |
   | Name        | MetricsImporter                |
   | Description | A test service account to      |
   |             | import Confluent Cloud metrics |
   |             | into our monitoring system     |
   +-------------+--------------------------------+
   

2. Make note of the ID field.

3. Add the MetricsViewer role binding to the service account:

   confluent iam rbac role-binding create --role MetricsViewer --principal User:sa-123abc
   

   Your output should resemble:

   +-----------+----------------+
   | Principal | User:sa-123abc |
   | Role      | MetricsViewer  |
   +-----------+----------------+
   

4. List the role bindings to confirm that the MetricsViewer role was created:

   confluent iam rbac role-binding list --principal User:sa-123abc
   

   Your output should resemble:

       Principal    | Email |     Role      | Environment | ...
   -----------------+-------+---------------+-------------+----
     User:sa-123abc |       | MetricsViewer |             |
   

5. List the existing service accounts:

   confluent iam service-account list
   

   Your output should resemble:

        ID     |              Name              |           Description
   ------------+--------------------------------+-----------------------------------
    sa-1a2b3c | test-account                   | for testing
    sa-112233 | ProactiveSupport.1614189731753 | SA for Proactive Support
    sa-aabbcc | KSQL.lksqlc-ab123              | SA for KSQL w/ ID lksqlc-ab123
              |                                | and Name ksqlDB_app_0
    ...
   

6. Create an API key and add it to the new service account:

   confluent api-key create --resource cloud --service-account sa-123abc
   

   Your output should resemble:

   It may take a couple of minutes for the API key to be ready.
   Save the API key and secret. The secret is not retrievable later.
   +---------+------------------------------------------------------------------+
   | API Key | 1234567ABCDEFGHI                                                 |
   | Secret  | ABCDEF123456.................................................... |
   +---------+------------------------------------------------------------------+
   

7. Save the API key and secret in a secure location.

Cloud Console

1. In the Administration menu (☰) of the Confluent Cloud Console, click Administration > API keys.

2. Click Add API key.

3. Click the My account or Service account tile to set the scope for the API key. If you selected Service account, choose the account name from the dropdown list or create a new account. Click Next.

4. Click Cloud resource management to ensure the key is a Cloud Cluster Key. Click Next.

5. Name your API key and optionally provide a description. Select Create API key.

6. The API key and secret are generated for the service account. You need this API key and secret to connect to the cluster, so be sure to safely store this information. Click Complete. The new service account with the API key and associated ACLs is created. When you return to the API access tab, you can view the newly created API key to confirm.

7. Return to Accounts & access in the administration menu, and in the Accounts tab, click Service accounts to view your service accounts.

8. Select the service account that you want to assign the MetricsViewer role to.

9. In the service account’s details page, click Access.

10. In the tree view, open the resource where you want the service account to have the MetricsViewer role.

11. Click Add role assignment and select the MetricsViewer tile. Click Save.

When you return to Accounts & access, you can view the resources for the organization, and also see that the service account you created has the MetricsViewer role binding.

Discover resources and metrics with the Metrics API

The Confluent Cloud Metrics API provides endpoints for programmatic discovery of available resources and their metrics. This resource and metric metadata is represented by descriptor objects.

Use the discovery endpoints to avoid hardcoding metric and resource names into client scripts.

Considerations

• The following examples use HTTPie and cURL.

• Install HTTPie using most common software package managers by following the documentation.

• cURL is a standard component of most operating systems, but if you don’t have cURL, you can install it by following the documentation.

Prerequisites

• Either HTTPie, cURL, or your favorite tool for making RESTful requests.

• API key resource-scoped for resource management.

• Confluent Cloud resources to monitor.

Discover available resources

A resource represents the entity against which metrics are collected, for example, a Kafka cluster, a Kafka connector, or a ksqlDB application.

Get a description of the available resources by sending a GET request to the descriptors/resources endpoint of the API:

HTTPie

http 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/resources' --auth '<API_KEY>:<SECRET>'

cURL

curl -X GET 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/resources' -u '<API_KEY>:<SECRET>'

This returns a JSON document describing the available resources to query and their labels.

Discover available metrics

Get a description of the available metrics by sending a GET request to the descriptors/metrics endpoint of the API:

HTTPie

http 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/metrics?resource_type=kafka' --auth '<API_KEY>:<SECRET>'

cURL

curl -X GET 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/metrics?resource_type=kafka' -u '<API_KEY>:<SECRET>'

Note

The resource_type query parameter is required to specify the type of resource for which to list metrics. The valid resource types can be determined using the /descriptors/resources endpoint.

This returns a JSON document describing the available metrics to query and their labels.

A human-readable list of the current metrics is available in the API Reference.

Metric data export and automated consumption

The Metrics API provides actionable operational metrics about your Confluent Cloud deployment, allowing for integration with external monitoring and observability platforms.

Prerequisites

• API key resource-scoped for resource management.

• Confluent Cloud resources to monitor.

• External monitoring tool such as Prometheus, Grafana, Datadog.

Export endpoint

Use the /export endpoint to export current metric values in OpenMetrics or Prometheus format, suitable for import into an external monitoring system. The /export endpoint returns the single most recent data point for each metric, for each distinct combination of labels. For more information, see the /export endpoint in the Metrics API reference.

To use the export endpoint:

1. Create a Confluent Cloud API key and secret with the MetricsViewer role.

2. Specify the target resource IDs, such as Kafka cluster IDs, using query parameters to limit the metrics to only the resources you want to monitor.

3. Set up an external monitoring tool such as Prometheus to initiate a periodic GET request, or a scrape, to the /export endpoint.

   • The endpoint returns a payload in an OpenMetrics-compatible format, which your external tool can immediately import, store, and use for visualization and alerting.

Discovery endpoint

Use the /discovery endpoint to configure your monitoring tool to dynamically discover and scrape metrics from all authorized Confluent Cloud resources. For more information, see the /discovery endpoint in the Metrics API reference.

To use the discovery endpoint:

1. Create a Confluent Cloud API key and secret with the MetricsViewer role.

2. Configure a Prometheus scrape job with the /discovery endpoint URL and the API key and secret.

   • When the Prometheus server connects to the /discovery endpoint, it receives a Prometheus-compatible list of scrape targets. This eliminates the need to mention resource IDs in the scraping-configuration file.

   • Prometheus automatically uses the response to the /discovery endpoint to dynamically generate, or relabel, a list of active target URLs for its metric scraping jobs.

   • If a new resource is created, such as a new Kafka cluster, Prometheus automatically adds the new target on its next scrape cycle, eliminating the need for manually adding new resources in the scraping-configuration file.

   To learn how to configure Prometheus HTTP-based service discovery, see http_sd_config in the Prometheus documentation.

KIP-714 client metrics

With the implementation of KIP-714, Kafka clients can now push selected metrics directly to Confluent Cloud brokers. This enhancement improves observability by allowing client metrics to be centrally available through the Confluent Cloud Metrics API.

Monitor client metrics collected by Confluent Cloud

Based on KIP-714, Confluent Cloud pushes and makes available the following client metric for monitoring: Confluent Cloud exposes the Kafka metric org.apache.kafka.producer.node.request.latency.avg as io.confluent.kafka.server/producer_latency_avg_milliseconds, which reports the average produce request latency.

Throttled clients metric

The io.confluent.kafka.server/client_limit_milliseconds metric reports the average throttle time, in milliseconds, applied to a principal when a quota is violated. Quotas can come from Confluent-managed cluster limits or from user-defined client quotas. Use this metric to identify which principals are being throttled, the type of quota that was violated, and the reason for the throttle, without needing client-side instrumentation. This metric is available for all Confluent Cloud cluster types.

Response fields

Field	Description
value	Average throttle time applied to the principal, in milliseconds.
resource.kafka.id	The Kafka cluster ID, for example, lkc-xxxxx.
metric.principal_id	The service account ID being throttled, for example, u-23agh7.
metric.violated_limit	The type of quota that was violated. Possible values: produce_throughput_quota: Produce throughput quota. fetch_throughput_quota: Fetch throughput quota. request_rate_quota: Request rate quota. For the underlying cause of the throttle in each case, see the metric.reason field.
metric.reason	The underlying reason for the throttle. Possible values: cluster_quota_violation: Total cluster traffic exceeded the cluster-level quota set by Confluent. You can manually increase cluster limits for Dedicated clusters by increasing your CKU count. eCKU limits automatically scale up to the max eCKU; to increase beyond this limit, file a support ticket. principal_quota_violation: Traffic exceeded a user-defined principal-level quota set by an administrator. To resolve this, reduce the application workload or redistribute it across additional resources. skew_traffic: Uneven traffic on specific partitions caused throttling even though total traffic did not exceed the cluster quota. This is typically a hot partition. Reconsider the key and partition strategy for the topic. scaling_out: Transient throttling during cluster scale-up, which returns to normal after scaling completes. other: Reserved for other reasons.

For more information about configuring client quotas, see Multi-Tenancy and Client Quotas on Confluent Cloud.

Related content

• Run Example Queries for the Metrics API

• Podcast: Multi-Cloud Monitoring and Observability with the Metrics API ft. Dustin Cote

• Blog: Bringing Your Own Monitoring (BYOM) with Confluent Cloud

• Build Streaming Applications

• Metrics Reference

• MetricsViewer