# Decommission an LKC Decommission a Logical Kafka Cluster (LKC) to permanently delete it from a multi-tenant Physical Kafka Cluster (PKC) and free its capacity. Platform operators perform this procedure when they no longer need an LKC. ## Prerequisites Before you begin, ensure you have the following: - `kubectl` access to delete the `LogicalKafkaCluster` custom resource (CR) and the ecosystem-component, `ConfluentRolebinding`, and `KafkaTopic` CRs that reference the LKC. - The `KafkaRestClass` that you used to create the LKC. ## Step 1: Prepare the LKC for decommissioning Notify the `TenantAdmin` that you plan to decommission their LKC, and give them enough lead time to complete the following before you continue: - Export any data their team needs to keep. Confluent Private Cloud doesn’t back up an LKC’s topics before deleting them, so exporting is the LKC owner’s own responsibility. Cluster Linking can mirror the LKC’s topics to another cluster first. For more information, see the Migrate Existing Clusters to Multi-Tenant LKCs page. After mirroring completes, promote or fail over the mirror topics on the destination cluster before you continue. - Delete every cluster link that uses this LKC as a source or destination, including any link that mirrors the LKC’s topics to another cluster. After the LKC is deactivated, its bootstrap URL and credentials no longer work, so you can’t remove a link through the LKC: ```bash kafka-cluster-links --bootstrap-server \ --command-config --list kafka-cluster-links --bootstrap-server \ --command-config --delete \ --link ``` - Export any schemas, connector configurations, or ksqlDB queries from ecosystem components dedicated to this LKC. [Delete dependent resources](#multi-tenancy-decommission-step-2) deletes these components. - Stop, or redirect elsewhere, any producer and consumer applications that still connect to this LKC. ## Step 2: Delete dependent resources Before you delete the LKC, delete the following. CFK doesn’t clean these up automatically: - Any Kafka Connect, Schema Registry, ksqlDB, or REST Proxy CR dedicated to this LKC. For example, `kubectl delete connect -n `. For more information about ecosystem components, see [Deploy Ecosystem Components on a Multi-Tenant PKC](ecosystem-components.md#multi-tenancy-ecosystem-components). - Any `ConfluentRolebinding` or `KafkaTopic` CR that references this LKC from a Kubernetes namespace other than the LKC’s own. When you delete the LKC, CFK automatically deletes any `ConfluentRolebinding` or `KafkaTopic` CR in the LKC’s own namespace, such as the `TenantAdmin` binding from [Step 7](onboard-a-tenant.md#multi-tenancy-onboard-step-7) of [Onboard an LKC to a Multi-Tenant Physical Kafka Cluster](onboard-a-tenant.md#multi-tenancy-onboard-a-tenant) and developer bindings from [Step 1](onboard-a-developer.md#multi-tenancy-onboard-developer-step-1) of [Onboard a Developer to an LKC](onboard-a-developer.md#multi-tenancy-onboard-a-developer). This automatic cleanup doesn’t extend to a role binding or topic CR in a different namespace. To find these CRs, run: ```bash kubectl get confluentrolebinding,kafkatopic -A \ -o custom-columns='KIND:.kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name,REF_KIND:.spec.kafkaClusterRef.kind,REF_NAME:.spec.kafkaClusterRef.name,REF_NAMESPACE:.spec.kafkaClusterRef.namespace' ``` Delete each result whose reference names this LKC but whose own `NAMESPACE` column differs from the LKC’s, with `kubectl delete -n `. ## Step 3: Delete the `LogicalKafkaCluster` CR Run the following command, where `` is the LKC’s Kubernetes object name, not the LKC display name. For the difference between the two, see [Step 3](onboard-a-tenant.md#multi-tenancy-onboard-step-3) of [Onboard an LKC to a Multi-Tenant Physical Kafka Cluster](onboard-a-tenant.md#multi-tenancy-onboard-a-tenant): ```bash kubectl delete logicalkafkacluster -n ``` This command returns when deletion completes, which normally takes seconds. While it’s in progress, `status.state` reads `DELETE-REQUESTED`. If the command doesn’t complete, see [Troubleshooting](#multi-tenancy-decommission-troubleshooting). ## Step 4: Wait for broker-side teardown Deleting the LKC CR doesn’t immediately delete the LKC’s topics and ACLs. The broker removes them in the background, which can take several minutes, and longer for an LKC with many topics. Wait a few minutes before you [verify the decommissioning](#multi-tenancy-decommission-verification). ## Verification - Confirm that `kubectl get logicalkafkacluster -n ` returns a not found error, which shows that [deleting the LogicalKafkaCluster CR](#multi-tenancy-decommission-step-3) completed. - Confirm that the LKC no longer appears in Confluent Control Center after you [delete the LogicalKafkaCluster CR](#multi-tenancy-decommission-step-3). - Using a principal that’s still valid, such as a platform operator identity rather than one you revoked in [Prepare the LKC for decommissioning](#multi-tenancy-decommission-step-1), confirm that a client connecting to the LKC’s former bootstrap URL can’t connect. For more information about connection failures, see the [Connection and authentication errors](onboard-a-developer.md#multi-tenancy-onboard-developer-connection-errors) section of [Onboard a Developer to an LKC](onboard-a-developer.md#multi-tenancy-onboard-a-developer). ## Troubleshooting ### `kubectl delete` doesn’t complete By default, `kubectl delete logicalkafkacluster` blocks your terminal until deletion completes, which [Delete the LogicalKafkaCluster CR](#multi-tenancy-decommission-step-3) says normally takes seconds. If your terminal is still blocked after that, open a second terminal and check `status.state` instead of waiting: ```bash kubectl get logicalkafkacluster -n \ -o jsonpath='{.status.state}' ``` While a failure persists, `status.state` can alternate between `ERROR` and `DELETE-REQUESTED` as CFK retries, so don’t rely on its value. Run `kubectl get logicalkafkacluster -n -o yaml` and check the CR’s status conditions for a `DeleteFailed` condition and its message. If there’s no `DeleteFailed` condition, the deletion is still in progress, so give it a little longer. Common causes of a failure include: - CFK can’t reach Metadata Service (MDS), or its `KafkaRestClass` doesn’t have a resolvable endpoint. Confirm that MDS is reachable and that the `KafkaRestClass` CFK uses is configured correctly, then wait. CFK retries automatically, so you don’t need to run `kubectl delete` again. - The `KafkaRestClass` CFK uses doesn’t authenticate as a principal with the `OrganizationAdmin` role on the PKC, and MDS rejects the deactivation request. Grant the `OrganizationAdmin` role on the PKC to that principal. CFK retries automatically, so you don’t need to run `kubectl delete` again. If neither applies and `status.state` stays at `ERROR`, contact [Confluent Support](https://support.confluent.io) before you take further action. ### Orphaned dependent resources If you deleted the LKC before completing [Delete dependent resources](#multi-tenancy-decommission-step-2), delete the remaining dedicated Kafka Connect, Schema Registry, ksqlDB, and REST Proxy CRs, and any cross-namespace `ConfluentRolebinding` or `KafkaTopic` CRs, individually. Deleting them after the LKC is already gone doesn’t cause an error. ### Stale cluster links If you deleted the LKC before completing [Prepare the LKC for decommissioning](#multi-tenancy-decommission-step-1), deactivating the LKC moved any cluster link that used it as a source or destination into a failed state, without removing the link record itself. The LKC’s own bootstrap URL and credentials no longer work, so you can’t remove the link the way [Step 1: Prepare the LKC for decommissioning](#multi-tenancy-decommission-step-1) describes. Contact [Confluent Support](https://support.confluent.io/) for help removing the stale link record. ## Related content - [Onboard an LKC to a Multi-Tenant Physical Kafka Cluster](onboard-a-tenant.md#multi-tenancy-onboard-a-tenant) - [Onboard a Developer to an LKC](onboard-a-developer.md#multi-tenancy-onboard-a-developer) - [Monitor and Manage Per-LKC Resource Quotas](manage-tenant-quotas.md#multi-tenancy-manage-tenant-quotas) - Manage Tenant Access with RBAC - Deploy Ecosystem Components with Multi-Tenant LKCs