# Monitor and Manage Per-LKC Resource Quotas Monitor how a Logical Kafka Cluster (LKC) uses its Physical Kafka Cluster (PKC), and update, increase, or troubleshoot its quotas and limits in Confluent Private Cloud. Platform operators perform these tasks on a running LKC. ## Prerequisites Before you begin, ensure you have the following: - A provisioned LKC. For more information, see [Onboard an LKC to a Multi-Tenant Physical Kafka Cluster](onboard-a-tenant.md#multi-tenancy-onboard-a-tenant). - The LKC’s Kubernetes object name and namespace. ## Monitor quota usage Track each LKC’s usage against its quotas to decide when to change a quota or add capacity. ### Use Confluent Control Center Confluent Control Center shows only the LKCs that your role bindings cover. With a PKC-scoped role binding, such as `OrganizationAdmin`, the **Quota Overview** page compares usage across every LKC on the PKC. With an LKC-scoped role binding, such as `TenantAdmin`, each LKC’s overview page has a **Quota allocation** panel. For more information, see the Confluent Control Center documentation. ### Query metrics in Prometheus On a multi-tenant PKC, broker metrics carry the following tags, which identify the LKC: | Tag | Value | |----------------|---------------------------------------------------------------------------| | `tenant` | The LKC ID, such as `lkc-000000`. | | `topic` | The topic name prefixed with the LKC ID, such as
`lkc-000000_orders`. | | `tenant_topic` | The topic name without the prefix, such as `orders`. | Filter or group by the `tenant` tag to measure one LKC or to compare LKCs. The following metrics measure LKC usage: | Metric | Description | |--------------------------------------------------------------------------|------------------------------------------| | `io_confluent_kafka_server_broker_topic_bytes_in_rate_1_min` | Bytes produced per second. | | `io_confluent_kafka_server_broker_topic_bytes_out_rate_1_min` | Bytes consumed per second. | | `io_confluent_kafka_server_tenant_active_authenticated_connection_count` | Active authenticated client connections. | For example, the following query returns the 95th percentile (p95) production rate of one LKC over the last hour. This is the value that the **Quota Overview** page shows: ```text quantile_over_time(0.95, (sum(io_confluent_kafka_server_broker_topic_bytes_in_rate_1_min{kafka_cluster_id="", tenant=""}))[1h:1m]) ``` Replace the placeholders with your values: - ``: The Kafka cluster ID of the PKC. - ``: The LKC ID. For instructions on collecting Confluent Platform metrics with Prometheus and Grafana when you deploy with Confluent for Kubernetes (CFK), see [Monitor Confluent Platform with Confluent for Kubernetes](https://docs.confluent.io/operator/current/co-monitor-cp.html). #### NOTE Broker log lines aren’t tagged by LKC, so you can’t filter broker logs for a single LKC. Use the `tenant` tag on metrics instead. Authorization audit events hide the tenant prefix. For more information, see [Audit log content](../../security/compliance/audit-logs/audit-logs-concepts.md#audit-log-content). ### Set up alerts A logical cluster trigger in Confluent Control Center monitors one or more LKCs. The **Logical Cluster** component type appears only after you enable multi-tenancy in Confluent Control Center. Select **Logical Cluster** as the component type, select the LKCs to monitor, and then choose one of the following metrics: **Bytes in**, **Bytes out**, **Connections**, **Partitions**, or **Under replicated partitions**. A trigger compares a metric with a fixed value that you enter. It doesn’t reference the LKC’s configured quota, and no trigger type alerts on quota usage. To alert before an LKC reaches its limit, set the value to a fraction of the quota. For example, if an LKC’s `ingressByteRate` is 50 MiB/s, set a **Bytes in** trigger to 80% of that value, in bytes per second. If you change an LKC’s quota, update its triggers to match. ## Manage LKC quotas If the usage you reviewed in [Monitor quota usage](#multi-tenancy-manage-quotas-monitor) shows that an LKC’s quotas or limits must change, complete the following steps. ### Step 1: Update quotas and limits on the LKC manifest The `LogicalKafkaCluster` custom resource (CR) enforces each field under `spec.quotas` and `spec.limits` as follows: | Field | Enforcement | |------------------------------------|-----------------| | `spec.quotas.ingressByteRate` | Soft: throttled | | `spec.quotas.egressByteRate` | Soft: throttled | | `spec.quotas.brokerRequestThreads` | Soft: throttled | | `spec.limits.maxPartitions` | Hard: rejected | | `spec.limits.maxMessageSizeBytes` | Hard: rejected | For the full field descriptions, see [Configure resource quotas and limits](onboard-a-tenant.md#multi-tenancy-onboard-step-2) of [Onboard an LKC to a Multi-Tenant Physical Kafka Cluster](onboard-a-tenant.md#multi-tenancy-onboard-a-tenant). To change an existing LKC’s quotas or limits, edit the `spec.quotas` and `spec.limits` fields on the LKC directly: ```bash kubectl edit logicalkafkacluster -n ``` The following table lists the default for each omitted field: | Omitted field | Default | |----------------------------------------|---------------------------------------------------------------------------------------------| | `ingressByteRate` and `egressByteRate` | No limit for the LKC. Only the PKC’s broker-wide limits apply,
and all LKCs share them. | | `brokerRequestThreads` | `2.5` threads | | `maxPartitions` | The broker’s default limit, `4096` partitions | | `maxMessageSizeBytes` | The broker’s default maximum | Lowering `spec.limits.maxPartitions` below an LKC’s current total partition count doesn’t delete existing partitions. After you lower the limit, the broker rejects requests to create topics or add partitions until the LKC’s partition count plus the requested partitions is at or below the new limit. ### Request and approve a quota increase 1. The `TenantAdmin` requests a quota increase through the escalation path you shared with them in [Provide connection details](onboard-a-tenant.md#multi-tenancy-onboard-step-8) of [Onboard an LKC to a Multi-Tenant Physical Kafka Cluster](onboard-a-tenant.md#multi-tenancy-onboard-a-tenant). 2. Evaluate the request against the remaining capacity of the PKC, using the capacity-planning baseline established when you deployed the PKC. 3. If the PKC has enough unused capacity, apply the new values by following [Update quotas and limits on the LKC manifest](#multi-tenancy-manage-quotas-step-1). If it doesn’t, scale the PKC, and then apply the new values. ### Verify the change - Confirm that the change took effect: ```bash kubectl get logicalkafkacluster -n -o yaml ``` Compare the returned `spec.quotas` and `spec.limits` against the values you applied. - Check the client metrics of the LKC’s applications to observe quota enforcement. When an LKC exceeds `ingressByteRate` or `egressByteRate`, its producers report a nonzero `produce-throttle-time-avg` or `produce-throttle-time-max`, and its consumers report a nonzero `fetch-throttle-time-avg` or `fetch-throttle-time-max`. The broker throttles the clients. It doesn’t return connection errors or reject requests. - Confirm that a request to create or increase partitions beyond `maxPartitions` is rejected immediately, with an error that identifies the exceeded limit. - Confirm that the broker rejects a request to create or alter a topic with a `max.message.bytes` value above `maxMessageSizeBytes`. - To view an LKC’s usage against its configured limits, see [Monitor quota usage](#multi-tenancy-manage-quotas-monitor). ## Troubleshooting ### Diagnose why an LKC’s quota isn’t taking effect When a `TenantAdmin` reports that a quota or limit doesn’t seem to take effect, compare the values on the LKC with the values you applied: ```bash kubectl get logicalkafkacluster -n -o yaml ``` If a field is missing from `spec.quotas` or `spec.limits`, the LKC uses the default for that field. For the defaults, see the table of omitted-field defaults in [Update quotas and limits on the LKC manifest](#multi-tenancy-manage-quotas-step-1). Then review the LKC’s metrics, as described in [Monitor quota usage](#multi-tenancy-manage-quotas-monitor). ### Investigate a noisy-neighbor LKC Follow this sequence to find and act on a noisy-neighbor LKC, one that’s degrading performance for other LKCs on the same PKC: 1. Check the client latency metrics of the applications in the affected LKCs. Elevated throttle time alone isn’t the symptom, because it means an LKC is exceeding its own quota and the quota is working. 2. Query the per-LKC produce and fetch byte-rate metrics, filtered by the `tenant=` tag, to find the LKC using a disproportionate share of the PKC’s capacity. An LKC can degrade others without ever being throttled itself if its quotas are sized too high for the PKC’s actual capacity. 3. Open the PKC’s **Quota Overview** page in Confluent Control Center. Determine whether the LKC’s usage is approaching its configured quotas or whether the quotas themselves are set too high for the PKC’s actual capacity. 4. If every LKC’s usage is legitimate and the PKC is at capacity, scale the PKC. Otherwise, lower the noisy LKC’s `spec.quotas` by following [Update quotas and limits on the LKC manifest](#multi-tenancy-manage-quotas-step-1). 5. Set up an alert so that you’re notified the next time this happens, instead of relying on the dashboard. For more information, see [Monitor quota usage](#multi-tenancy-manage-quotas-monitor). ## Related content - [Onboard an LKC to a Multi-Tenant Physical Kafka Cluster](onboard-a-tenant.md#multi-tenancy-onboard-a-tenant) - Deploy a Multi-Tenant Physical Kafka Cluster - Quota Overview page in Confluent Control Center - Tenant Management API in Metadata Service (MDS)