# Confluent Cloud Metrics The Confluent Cloud Metrics API is a REST API that returns time-series metrics for Apache Kafka® clusters, [connectors](../_glossary.md#term-connector), [Stream Governance](../_glossary.md#term-Stream-Governance) components, and Apache Flink® [compute pools](../_glossary.md#term-stream-processing). Use it to query: * Throughput * Consumer lag * Partition count * Request rates * Other operational metrics Metrics are also available through integrations with Datadog, Prometheus, Grafana, and other [third-party monitoring platforms](third-party-integration.md#ccloud-integrate-with-3rd-party-monitoring). For ready-to-use queries covering these resource types, see [Run Example Queries for the Metrics API](metrics-api-examples.md#metrics-examples). ## Metrics quick start Use this quick start to create an API key and run example queries against the Metrics API. **Considerations** - You must use an API key resource-scoped for resource management to communicate with the Metrics API. - API keys resource-scoped for a Kafka cluster cause an authentication error. - Metrics in Confluent Cloud are available either through first-class integrations with [third-party monitoring providers](third-party-integration.md#ccloud-integrate-with-3rd-party-monitoring) or by directly querying the Confluent Cloud Metrics API. - To reduce the operational burden of monitoring, use an integration with [third-party monitoring providers](third-party-integration.md#ccloud-integrate-with-3rd-party-monitoring). - The Metrics API’s IP addresses are not static. For more information, see [IP addresses for public endpoints](../networking/overview.md#cloud-ip-addresses-not-static). **Prerequisites** - API key resource-scoped for resource management. - Confluent Cloud resources to monitor. **Get started**: Create an API key and run example queries. For more information, see [Create an API key to authenticate to the Metrics API](#ccloud-add-metricsviewer-role-to-new-service-account) and [Run Example Queries for the Confluent Cloud Metrics API](metrics-api-examples.md#metrics-examples). #### SEE ALSO For an example that showcases how to monitor a Kafka client application and Confluent Cloud metrics, and steps through various failure scenarios to show metrics results, see the [Observability for Kafka Clients to Confluent Cloud](ccloud-observability.md#ccloud-observability-index). ## Create an API key to authenticate to the Metrics API To securely access Confluent Cloud metrics programmatically or through third-party monitoring tools, you need both the [MetricsViewer](../security/access-control/rbac/predefined-rbac-roles.md#metricsviewer-role) role for authorization and an [API key](https://docs.confluent.io/confluent-cli/current/command-reference/api-key/confluent_api-key_create.html) for authentication. The MetricsViewer role provides service account access to the Metrics API for all clusters in an organization. This role also enables service accounts to import metrics into third-party metrics platforms. To create an API key to authenticate to the Metrics API: ### Confluent CLI Run the following commands to add a role binding for MetricsViewer to a new service account. Remember to log in with the `confluent login` command first. 1. Create the service account: ```bash confluent iam service-account create MetricsImporter --description "A test service account to import Confluent Cloud metrics into our monitoring system" ``` Your output should resemble: ```none +-------------+--------------------------------+ | ID | sa-123abc | | Name | MetricsImporter | | Description | A test service account to | | | import Confluent Cloud metrics | | | into our monitoring system | +-------------+--------------------------------+ ``` 2. Make note of the ID field. 3. Add the MetricsViewer role binding to the service account: ```bash confluent iam rbac role-binding create --role MetricsViewer --principal User:sa-123abc ``` Your output should resemble: ```none +-----------+----------------+ | Principal | User:sa-123abc | | Role | MetricsViewer | +-----------+----------------+ ``` 4. List the role bindings to confirm that the MetricsViewer role was created: ```bash confluent iam rbac role-binding list --principal User:sa-123abc ``` Your output should resemble: ```none Principal | Email | Role | Environment | ... -----------------+-------+---------------+-------------+---- User:sa-123abc | | MetricsViewer | | ``` 5. List the existing service accounts: ```bash confluent iam service-account list ``` Your output should resemble: ```none ID | Name | Description ------------+--------------------------------+----------------------------------- sa-1a2b3c | test-account | for testing sa-112233 | ProactiveSupport.1614189731753 | SA for Proactive Support sa-aabbcc | KSQL.lksqlc-ab123 | SA for KSQL w/ ID lksqlc-ab123 | | and Name ksqlDB_app_0 ... ``` 6. Create an API key and add it to the new service account: ```bash confluent api-key create --resource cloud --service-account sa-123abc ``` Your output should resemble: ```none It may take a couple of minutes for the API key to be ready. Save the API key and secret. The secret is not retrievable later. +---------+------------------------------------------------------------------+ | API Key | 1234567ABCDEFGHI | | Secret | ABCDEF123456.................................................... | +---------+------------------------------------------------------------------+ ``` 7. Save the API key and secret in a secure location. ### Cloud Console 1. In the Administration menu (☰) of the Confluent Cloud Console, click **Administration** > **API keys**. 2. Click **Add API key**. 3. Click the **My account** or **Service account** tile to set the scope for the API key. If you selected **Service account**, choose the account name from the dropdown list or create a new account. Click **Next**. 4. On the **Resource scope** page, click **Cloud resource management**. Click **Next**. #### IMPORTANT The resource scope must be **Cloud resource management**. An API key scoped to a Kafka cluster causes an authentication error when used with the Metrics API. 5. Name your API key and optionally provide a description. Select **Create API key**. 6. The API key and secret are generated for the service account. You need this API key and secret to authenticate to the Metrics API, so be sure to safely store this information. Click **Complete**. When you return to the **API keys** page, you can view the newly created API key to confirm. 7. Return to **Accounts & access** in the administration menu, and in the **Accounts** tab, click **Service accounts** to view your service accounts. 8. Select the service account that you want to assign the MetricsViewer role to. 9. In the service account’s details page, click **Access**. 10. In the tree view, open the resource where you want the service account to have the MetricsViewer role. 11. Click **Add role assignment** and select the **MetricsViewer** tile. Click **Save**. When you return to **Accounts & access**, you can view the resources for the organization, and also see that the service account you created has the MetricsViewer role binding. ## Discover resources and metrics with the Metrics API The Confluent Cloud Metrics API provides endpoints for programmatic discovery of available resources and their metrics. This resource and metric metadata is represented by `descriptor` objects. A descriptor is a JSON object that describes an available resource or metric and its labels. Use the discovery endpoints to avoid hardcoding metric and resource names into client scripts. **Considerations** - The following examples use [HTTPie](https://httpie.org/doc#installation) and [cURL](https://curl.se/). - Install HTTPie using most common software package managers by following [the documentation](https://httpie.org/doc#installation). - cURL is a standard component of most operating systems, but if you don’t have cURL, you can install it by following [the documentation](https://curl.se/docs/). **Prerequisites** - Either HTTPie, cURL, or your favorite tool for making RESTful requests. - API key resource-scoped for resource management. - Confluent Cloud resources to monitor. ### Discover available resources A resource represents the entity against which metrics are collected, for example, a Kafka cluster, a Kafka connector, or a ksqlDB application. Get a description of the available resources by sending a `GET` request to the `descriptors/resources` endpoint of the API: ### HTTPie ```bash http 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/resources' --auth ':' ``` ### cURL ```bash curl -X GET 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/resources' -u ':' ``` This returns a JSON document describing the available resources to query and their labels. ### Discover available metrics Get a description of the available metrics by sending a `GET` request to the `descriptors/metrics` endpoint of the API: ### HTTPie ```bash http 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/metrics?resource_type=kafka' --auth ':' ``` ### cURL ```bash curl -X GET 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/metrics?resource_type=kafka' -u ':' ``` #### NOTE The `resource_type` query parameter is required to specify the type of resource for which to list metrics. The valid resource types can be determined using the `/descriptors/resources` endpoint. This returns a JSON document describing the available metrics to query and their labels. A human-readable list of the current metrics is available in the [API Reference](https://api.telemetry.confluent.cloud/docs/descriptors/datasets/cloud). ## Metric data export and automated consumption The Metrics API provides actionable operational metrics about your Confluent Cloud deployment, allowing for integration with external monitoring and observability platforms. **Prerequisites** - API key resource-scoped for resource management. - Confluent Cloud resources to monitor. - External monitoring tool such as Prometheus, Grafana, Datadog. ### Metrics API export endpoint Use the `/export` endpoint to export current metric values in [OpenMetrics](https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md) or [Prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format) format, suitable for import into an external monitoring system. The `/export` endpoint returns the single most recent data point for each metric, for each distinct combination of labels. For more information, see the `/export` endpoint in the [Metrics API reference](https://api.telemetry.confluent.cloud/docs?session_ref=direct&url_ref=https%3A%2F%2Fdocs.confluent.io%2Fcloud%2Fcurrent%2Fmonitoring%2Fmetrics-api.html#tag/Version-2/paths/~1v2~1metrics~1%7Bdataset%7D~1query/post). To use the export endpoint: : 1. Create a Confluent Cloud API key and secret with the MetricsViewer role, which grants service-account access to the Metrics API. 2. Specify the target resource IDs, such as Kafka cluster IDs, using query parameters to limit the metrics to only the resources you want to monitor. 3. Set up an external monitoring tool such as Prometheus to initiate a periodic `GET` request, or a *scrape*, to the `/export` endpoint. - The endpoint returns a payload in an OpenMetrics-compatible format, which your external tool can immediately import, store, and use for visualization and alerting. ### Metrics API discovery endpoint Use the `/discovery` endpoint to configure your monitoring tool to dynamically discover and scrape metrics from all authorized Confluent Cloud resources. For more information, see the `/discovery` endpoint in the [Metrics API reference](https://api.telemetry.confluent.cloud/docs#tag/Version-2/paths/~1v2~1metrics~1%7Bdataset%7D~1discovery/get). To use the discovery endpoint: : 1. Create a Confluent Cloud API key and secret with the MetricsViewer role, which grants service-account access to the Metrics API. 2. Configure a Prometheus scrape job with the `/discovery` endpoint URL and the API key and secret. - When the Prometheus server connects to the `/discovery` endpoint, it receives a Prometheus-compatible list of scrape targets. This eliminates the need to mention resource IDs in the scraping-configuration file. - Prometheus automatically uses the response to the `/discovery` endpoint to dynamically generate, or relabel, a list of active target URLs for its metric scraping jobs. - If a new resource is created, such as a new Kafka cluster, Prometheus automatically adds the new target on its next scrape cycle, eliminating the need for manually adding new resources in the scraping-configuration file.
To learn how to configure Prometheus HTTP-based service discovery, see [http_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) in the Prometheus documentation. ## Client-side metrics pushed to Confluent Cloud Kafka clients can push selected metrics directly to Confluent Cloud brokers, enabled by [Kafka Improvement Proposal (KIP) 714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability). This enhancement improves observability by allowing client metrics to be centrally available through the Confluent Cloud Metrics API. ### Monitor client metrics collected by Confluent Cloud Confluent Cloud exposes the Kafka metric `org.apache.kafka.producer.node.request.latency.avg` as `io.confluent.kafka.server/producer_latency_avg_milliseconds`. This metric reports the average produce request latency. ## Throttled clients metric The `io.confluent.kafka.server/client_limit_milliseconds` metric reports the average throttle time, in milliseconds, applied to a principal when a quota is violated. Quotas can come from Confluent-managed cluster limits or from user-defined [client quotas](../clusters/client-quotas.md#client-quotas). Use this metric to identify which principals are being throttled, the type of quota that was violated, and the reason for the throttle, without needing client-side instrumentation. This metric is available for all Confluent Cloud cluster types. ### Throttled clients metric response fields | Field | Description | |-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `value` | Average throttle time applied to the principal, in milliseconds. | | `resource.kafka.id` | The Kafka cluster ID, for example, `lkc-xxxxx`. | | `metric.principal_id` | The service account ID being throttled, for example, `u-23agh7`. | | `metric.violated_limit` | The type of quota that was violated. Possible values:

- `produce_throughput_quota`: Produce throughput quota.
- `fetch_throughput_quota`: Fetch throughput quota.
- `request_rate_quota`: Request rate quota.

For the underlying cause of the throttle in each case, see the
`metric.reason` field. | | `metric.reason` | The underlying reason for the throttle. Possible values:

- `cluster_quota_violation`: Total cluster traffic exceeded the
cluster-level quota set by Confluent. You can manually increase
cluster limits for Dedicated clusters by increasing your
Confluent Unit for Kafka (CKU) count. [eCKU](../billing/billing-dimensions.md#e-cku-definition) limits automatically
scale up to the [max eCKU](../clusters/cluster-types.md#min-max-ecku); to increase beyond
this limit, file a [support ticket](https://support.confluent.io/).
- `principal_quota_violation`: Traffic exceeded a user-defined
principal-level quota set by an administrator. To resolve this, reduce the
application workload or redistribute it across additional resources.
- `skew_traffic`: Uneven traffic on specific partitions caused
throttling even though total traffic did not exceed the cluster
quota. This is typically a hot partition. Reconsider the key and
partition strategy for the topic.
- `scaling_out`: Transient throttling during cluster scale-up, which
returns to normal after scaling completes.
- `other`: Reserved for other reasons. | For more information about configuring client quotas, see [Multi-Tenancy and Client Quotas on Confluent Cloud](../clusters/client-quotas.md#client-quotas). ## Related content - [Run Example Queries for the Metrics API](metrics-api-examples.md#metrics-examples) - Podcast: [Multi-Cloud Monitoring and Observability with the Metrics API ft. Dustin Cote](https://podcasts.apple.com/us/podcast/multi-cloud-monitoring-observability-metrics-api-ft/id1401509765?i=1000461098902) - Blog: [Bringing Your Own Monitoring (BYOM) with Confluent Cloud](https://www.confluent.io/blog/bring-your-own-monitoring-with-confluent-cloud/) - [Build Streaming Applications](../client-apps/index.md#ccloud-best-practices) - [Metrics Reference](https://api.telemetry.confluent.cloud/docs/descriptors/datasets/cloud) - [MetricsViewer](../security/access-control/rbac/predefined-rbac-roles.md#metricsviewer-role)