Monitor and Manage Per-LKC Resource Quotas

Monitor how a Logical Kafka Cluster (LKC) uses its Physical Kafka Cluster (PKC), and update, increase, or troubleshoot its quotas and limits in Confluent Private Cloud. Platform operators perform these tasks on a running LKC.

Prerequisites

Before you begin, ensure you have the following:

A provisioned LKC. For more information, see Onboard an LKC to a Multi-Tenant Physical Kafka Cluster.
The LKC’s Kubernetes object name and namespace.

Monitor quota usage

Track each LKC’s usage against its quotas to decide when to change a quota or add capacity.

Use Confluent Control Center

Confluent Control Center shows only the LKCs that your role bindings cover. With a PKC-scoped role binding, such as OrganizationAdmin, the Quota Overview page compares usage across every LKC on the PKC. With an LKC-scoped role binding, such as TenantAdmin, each LKC’s overview page has a Quota allocation panel. For more information, see the Confluent Control Center documentation.

Query metrics in Prometheus

On a multi-tenant PKC, broker metrics carry the following tags, which identify the LKC:

Tag	Value
`tenant`	The LKC ID, such as `lkc-000000`.
`topic`	The topic name prefixed with the LKC ID, such as `lkc-000000_orders`.
`tenant_topic`	The topic name without the prefix, such as `orders`.

Filter or group by the tenant tag to measure one LKC or to compare LKCs. The following metrics measure LKC usage:

Metric	Description
`io_confluent_kafka_server_broker_topic_bytes_in_rate_1_min`	Bytes produced per second.
`io_confluent_kafka_server_broker_topic_bytes_out_rate_1_min`	Bytes consumed per second.
`io_confluent_kafka_server_tenant_active_authenticated_connection_count`	Active authenticated client connections.

For example, the following query returns the 95th percentile (p95) production rate of one LKC over the last hour. This is the value that the Quota Overview page shows:

quantile_over_time(0.95, (sum(io_confluent_kafka_server_broker_topic_bytes_in_rate_1_min{kafka_cluster_id="<pkc-id>", tenant="<lkc-id>"}))[1h:1m])

Replace the placeholders with your values:

<pkc-id>: The Kafka cluster ID of the PKC.
<lkc-id>: The LKC ID.

For instructions on collecting Confluent Platform metrics with Prometheus and Grafana when you deploy with Confluent for Kubernetes (CFK), see Monitor Confluent Platform with Confluent for Kubernetes.

Note

Broker log lines aren’t tagged by LKC, so you can’t filter broker logs for a single LKC. Use the tenant tag on metrics instead. Authorization audit events hide the tenant prefix. For more information, see Audit log content.

Set up alerts

A logical cluster trigger in Confluent Control Center monitors one or more LKCs. The Logical Cluster component type appears only after you enable multi-tenancy in Confluent Control Center. Select Logical Cluster as the component type, select the LKCs to monitor, and then choose one of the following metrics: Bytes in, Bytes out, Connections, Partitions, or Under replicated partitions.

A trigger compares a metric with a fixed value that you enter. It doesn’t reference the LKC’s configured quota, and no trigger type alerts on quota usage. To alert before an LKC reaches its limit, set the value to a fraction of the quota. For example, if an LKC’s ingressByteRate is 50 MiB/s, set a Bytes in trigger to 80% of that value, in bytes per second. If you change an LKC’s quota, update its triggers to match.

Manage LKC quotas

If the usage you reviewed in Monitor quota usage shows that an LKC’s quotas or limits must change, complete the following steps.

Step 1: Update quotas and limits on the LKC manifest

The LogicalKafkaCluster custom resource (CR) enforces each field under spec.quotas and spec.limits as follows:

Field	Enforcement
`spec.quotas.ingressByteRate`	Soft: throttled
`spec.quotas.egressByteRate`	Soft: throttled
`spec.quotas.brokerRequestThreads`	Soft: throttled
`spec.limits.maxPartitions`	Hard: rejected
`spec.limits.maxMessageSizeBytes`	Hard: rejected

For the full field descriptions, see Configure resource quotas and limits of Onboard an LKC to a Multi-Tenant Physical Kafka Cluster.

To change an existing LKC’s quotas or limits, edit the spec.quotas and spec.limits fields on the LKC directly:

kubectl edit logicalkafkacluster <lkc-name> -n <your-namespace>

Caution

Include every subfield of spec.quotas and spec.limits that you want to keep in your edit, not only the ones you’re changing. CFK overwrites both fields on every reconciliation and resets any subfield you omit to its default.

The following table lists the default for each omitted field:

Omitted field	Default
`ingressByteRate` and `egressByteRate`	No limit for the LKC. Only the PKC’s broker-wide limits apply, and all LKCs share them.
`brokerRequestThreads`	`2.5` threads
`maxPartitions`	The broker’s default limit, `4096` partitions
`maxMessageSizeBytes`	The broker’s default maximum

Lowering spec.limits.maxPartitions below an LKC’s current total partition count doesn’t delete existing partitions. After you lower the limit, the broker rejects requests to create topics or add partitions until the LKC’s partition count plus the requested partitions is at or below the new limit.

Request and approve a quota increase

The TenantAdmin requests a quota increase through the escalation path you shared with them in Provide connection details of Onboard an LKC to a Multi-Tenant Physical Kafka Cluster.
Evaluate the request against the remaining capacity of the PKC, using the capacity-planning baseline established when you deployed the PKC.
If the PKC has enough unused capacity, apply the new values by following Update quotas and limits on the LKC manifest. If it doesn’t, scale the PKC, and then apply the new values.

Verify the change

Confirm that the change took effect:
```
kubectl get logicalkafkacluster <lkc-name> -n <your-namespace> -o yaml
```
Compare the returned spec.quotas and spec.limits against the values you applied.
Check the client metrics of the LKC’s applications to observe quota enforcement. When an LKC exceeds ingressByteRate or egressByteRate, its producers report a nonzero produce-throttle-time-avg or produce-throttle-time-max, and its consumers report a nonzero fetch-throttle-time-avg or fetch-throttle-time-max. The broker throttles the clients. It doesn’t return connection errors or reject requests.
Confirm that a request to create or increase partitions beyond maxPartitions is rejected immediately, with an error that identifies the exceeded limit.
Confirm that the broker rejects a request to create or alter a topic with a max.message.bytes value above maxMessageSizeBytes.
To view an LKC’s usage against its configured limits, see Monitor quota usage.

Troubleshooting

Diagnose why an LKC’s quota isn’t taking effect

When a TenantAdmin reports that a quota or limit doesn’t seem to take effect, compare the values on the LKC with the values you applied:

kubectl get logicalkafkacluster <lkc-name> -n <your-namespace> -o yaml

If a field is missing from spec.quotas or spec.limits, the LKC uses the default for that field. For the defaults, see the table of omitted-field defaults in Update quotas and limits on the LKC manifest. Then review the LKC’s metrics, as described in Monitor quota usage.

Investigate a noisy-neighbor LKC

Follow this sequence to find and act on a noisy-neighbor LKC, one that’s degrading performance for other LKCs on the same PKC:

Check the client latency metrics of the applications in the affected LKCs. Elevated throttle time alone isn’t the symptom, because it means an LKC is exceeding its own quota and the quota is working.
Query the per-LKC produce and fetch byte-rate metrics, filtered by the tenant=<lkc-id> tag, to find the LKC using a disproportionate share of the PKC’s capacity. An LKC can degrade others without ever being throttled itself if its quotas are sized too high for the PKC’s actual capacity.
Open the PKC’s Quota Overview page in Confluent Control Center. Determine whether the LKC’s usage is approaching its configured quotas or whether the quotas themselves are set too high for the PKC’s actual capacity.
If every LKC’s usage is legitimate and the PKC is at capacity, scale the PKC. Otherwise, lower the noisy LKC’s spec.quotas by following Update quotas and limits on the LKC manifest.
Set up an alert so that you’re notified the next time this happens, instead of relying on the dashboard. For more information, see Monitor quota usage.