Track Usage by Team on Dedicated Clusters in Confluent Cloud

When you support multiple tenants on a Dedicated cluster, you might need to track usage by application to provide showbacks to internal customers for their consumption.

This document describes a model for tracking usage and implementing showbacks of Confluent Cloud Dedicated cluster costs based on Service Accounts.

Calculate monthly costs by team

To track usage by team, you assign each unique team or application its own service account. Then you use the Metrics API and filter results using the principal_id label to separate usage by service account. You track and sum this usage on a monthly basis, and use it to create a derived showback of costs for each service account.

Why use principals instead of topics

Topics are shared resources like clusters. The goal of showbacks is to divide the cost of shared resources across the teams responsible for creating the costs. The ideal way to do this is with principals. Each principal (a user or service account) should represent an application programmatically accessing Confluent Cloud. By breaking down billing metrics by principal, you can accurately assign costs like throughput to each application responsible for the costs.

Prerequisites

To accurately track usage by principal ID, you:

  • Must have similar retention times across all the topics

  • Can map service accounts to teams

  • Want to implement showbacks to teams by throughput usage

  • Must have enough access to a Confluent Cloud cluster to make queries and view billing information

Query for usage

Execute daily queries for the previous 24-hour interval, making sure the daily window is in the past.

Steps to determine usage by principal:

  1. Get the request bytes for a cluster daily by making a POST call to the Metrics API, filtered by principal ID. Store the data in a reliable location by month. For more details on this call, see the Metrics API Reference.

    Your request might look like the following:

    curl --location --request POST 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/query' \
--header 'Authorization: Basic <BASE-64-encoded-cloud-api-key-and-password>' \
--header 'Content-Type: application/json' \
--data-raw '{
"aggregations": [
  {
      "metric": "io.confluent.kafka.server/request_bytes"
  }
],
"filter": {
  "field": "resource.kafka.id",
  "op": "EQ",
  "value": "lkc-momo2"
},
"granularity": "P1D",
"group_by": [
  "metric.principal_id"
],
"intervals": [
    "2022-09-11T00:00:00-00:00/P1D"
],
"limit": 1000
}'

    Your response resembles the following. Store the returned value for each service account (this example shows one) on a daily basis. The returned value is in scientific notation, which you probably want to convert to decimal format to calculate a showback amount.

    {"data":[{"timestamp":"2022-09-11T00:00:00Z","value":4.69304195E8,"metric.principal_id":"sa-abcj5m"}]}

  2. Get the response bytes for a cluster daily by making a POST call to the Metrics API. Store the data by month in a reliable location. For more details on this call, see the Metrics API Reference.

    Your request to get all data for a cluster, sorted by principal ID, might look like the following:

    curl --location --request POST 'https://api.telemetry.confluent.cloud/v2/metrics/cloud/query' \
--header 'Authorization: Basic <BASE-64-encoded-cloud-api-key-and-password>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "aggregations": [
     {
         "metric": "io.confluent.kafka.server/response_bytes"
     }
 ],
 "filter": {
     "field": "resource.kafka.id",
     "op": "EQ",
     "value": "lkc-abc12"
 },
 "granularity": "P1D",
 "group_by": [
     "metric.principal_id"
 ],
 "intervals": [
     "2022-09-11T00:00:00-00:00/P1D"
 ],
 "limit": 1000
}'

    The response resembles the following. Store the value for each service account (this example shows one) on a daily basis. The returned value is in scientific notation, which you probably want to convert to decimal format to calculate a showback amount.

    {"data":[{"timestamp":"2022-09-11T00:00:00Z","value":2.44310325E8,"metric.principal_id":"sa-abcj5m"}]}

  3. At the end of a month, calculate the totals of all requests and response bytes, and sum the usage by principal_id, as follows:

    • Calculate the sum of the daily request bytes value for each principal_id for the month. Call this sum total_request_bytes<principal_id>.

    • Calculate the total sum of request bytes for all principals for the month. Call this value total_request_bytes.

    • Calculate the sum of the daily response bytes value for each principal_id for the month. Call these values total_response_bytes<principal_id>.

    • Calculate the total sum of all response bytes for all principals for the month. Call this value total_response_bytes.

Get the monthly bill for a cluster

The next step in determining showbacks is to get your monthly Confluent Cloud bill for the cluster.

  1. Use the Confluent Cloud Console to get your bill for a month. You can find Billing & payment on the Administration menu.

  2. On the Billing & payment page, use the drop-downs to select a month and an environment. Confluent separates charges for that environment by cluster.

  3. There might be several categories billed for the cluster. Add up these costs for the cluster you want to calculate showbacks for. For this example, this value is called total_billing.

    • Multiply total_billing for the cluster by 0.33. This is your request_bytes_cost.

    • Multiply total_billing for the cluster by 0.67. This is your response_bytes_cost. The ~2:1 ratio between response bytes weight and request bytes weight in the model reflects the higher cost of consumption in Confluent Cloud multi-availability-zone (AZ) clusters.

      Your calculations might look like the following:

      Month = August 2022

      • total_billing = $92,935

      • request_bytes_cost = $92,935 * 0.33 = $30,668.55

      • response_bytes_cost = $92,935 * 0.67 = $62,266.45

Calculate the monthly showbacks

Finally, use the monthly byte usage and Confluent Cloud billing data to calculate monthly showbacks.

  1. Calculate the showback for request and response bytes by principal ID.

    • total_request_cost_<principal_id> = total_billing * 0.33 * total_request_bytes<principal_id> / total_request_bytes

    • total_response_cost_<principal_id> = total_billing * 0.67 * total_response_bytes<principal_id> / total_response_bytes

  2. Add total_request_cost_<principal_id> and total_response_cost_<principal_id> from the previous step for each principal ID to calculate the total monthly showback for an internal customer or team.

Note

For clusters approaching one of the Confluent Unit for Kafka (CKU) limits, you can use the principal_id label on the following metrics to determine whether a particular team has outsized usage that should be considered in showbacks. This also helps prevent the need to scale up the Dedicated cluster unnecessarily. You can also use the cluster load metric to help determine if you are approaching maximum load on a cluster.

  • io.confluent.kafka.server/active_connection_count

  • io.confluent.kafka.server/request_count

  • io.confluent.kafka.server/successful_authentication_count

Related content