# Manage Multiple Kubernetes Clusters in Confluent Manager for Apache Flink Confluent Manager for Apache Flink® (CMF) supports managing Flink workloads across multiple Kubernetes clusters. A single CMF instance can register and manage additional clusters, deploy environments and applications on them, and monitor their health, all through the same REST API. CMF represents every Kubernetes cluster it manages as a `KubernetesCluster` resource, including the cluster where CMF itself is installed (the *hub* cluster). By default, CMF deploys Flink resources on this hub cluster. Multi-cluster support allows additional *spoke* clusters to be registered. Once registered, [environments](environments.md#cmf-environments) can be bound to any cluster, and Flink applications, compute pools, and SQL statements can be deployed on that cluster. This topic describes how to register, monitor, and decommission Kubernetes clusters in CMF. #### NOTE You can view, decommission, and recommission Kubernetes clusters with the REST API or the CMF UI. Clusters are registered only through kubeconfig files. ## Prerequisites Before registering a spoke cluster: 1. **Install the Flink Kubernetes Operator (FKO) on the spoke cluster.** The FKO and its Custom Resource Definitions (CRDs) (`FlinkDeployment`, `FlinkSessionJob`, `FlinkStateSnapshot`) must be present before the cluster is added to CMF. Registering a cluster before FKO is installed leads to an inconsistent state where the cluster appears connected but workload operations are not monitored correctly. To recover, install FKO on the cluster and then decommission and recommission the cluster via the API. CMF validates the FKO installation when a cluster is first registered and again whenever it reconnects after a `DISCONNECTED` transition. This validation is advisory — CMF logs a warning but does not block registration if the check fails. 2. **Place a kubeconfig file for the cluster in** CMF‘s kubeconfig directory (`/etc/cmf/kubeconfig` by default, configurable with `cmf.k8s.kubeconfigDirectory`). The filename must exactly match the cluster name used during registration. For example, a file named `my-cluster` registers the cluster under the name `my-cluster`. The filename must be a valid cluster name: lowercase letters, digits, and hyphens, starting with a letter, ending with a letter or digit, and at most 45 characters long. CMF does not register a file with any other name. If CMF runs in a Kubernetes pod, see [Mount kubeconfig files from a Kubernetes Secret](#cmf-k8s-kubeconfig-injection) for a Secret-backed mounting pattern. 3. **Ensure network and RBAC access.** CMF must be able to reach the spoke cluster’s API server, and the kubeconfig must grant CMF sufficient permissions to connect to the API server and monitor Flink resources on the cluster. ## Kubeconfig management CMF watches the kubeconfig directory for file changes at runtime, so clusters can be registered and deregistered without restarting CMF. Adding a kubeconfig file registers the cluster, and replacing a file, such as for credential rotation, re-initializes the connection to that cluster automatically. If the kubeconfig files are mounted from a Kubernetes Secret or ConfigMap, CMF does not detect changes to the contents of an existing file, so restart CMF after you update the Secret or ConfigMap. Removing the file of an `ACTIVE` cluster does not deregister it. The cluster stays registered and becomes `DISCONNECTED`. To deregister a cluster, [decommission](#cmf-k8s-decommission-cluster) it first, and then remove its kubeconfig file. CMF then deletes the cluster record. #### NOTE The maximum kubeconfig file size is 10 MB. For details on staging kubeconfig files in a containerized deployment, see [Mount kubeconfig files from a Kubernetes Secret](#cmf-k8s-kubeconfig-injection). For other kubeconfig settings, see the [configuration reference](#cmf-k8s-config-ref). ## Mount kubeconfig files from a Kubernetes Secret Because Confluent Manager for Apache Flink (CMF) runs in a container, you cannot durably place `kubeconfig` files on the pod filesystem by copying them in at runtime — files written with tools like `kubectl cp` are lost on pod restart, and staging files on a shared `ReadWriteMany` persistent volume so something outside the pod can write to it is discouraged. The recommended pattern is to store `kubeconfig` files in a Kubernetes Secret and volume-mount the Secret at `cmf.k8s.kubeconfigDirectory`. Each key in the Secret becomes a file inside the mounted directory, and CMF registers each file as a spoke cluster under the filename. The filename-as-cluster-name rule from the Prerequisites applies here — choose Secret keys that match the cluster names you intend to register. #### NOTE Although CMF supports `kubeconfig` files up to 10 MB, Kubernetes Secret objects are typically limited to about 1 MiB total size per object. If a `kubeconfig` is too large to fit in a Secret, reduce its size where possible (for example, avoid embedding large certificate or key data), or use an alternative delivery mechanism that places the `kubeconfig` file in `cmf.k8s.kubeconfigDirectory` without storing it in a Kubernetes Secret. 1. Create a Kubernetes Secret containing the `kubeconfig` files: ```bash kubectl create secret generic cmf-kubeconfigs \ --from-file=spoke-us-west=/local/path/to/us-west.kubeconfig \ --from-file=spoke-eu-central=/local/path/to/eu-central.kubeconfig \ -n ``` The key on the left of each `=` is the filename that appears in the mounted directory and the cluster name CMF registers. The path on the right is the source `kubeconfig` on your workstation. Do not append an extension such as `.kubeconfig` to the left-hand side. A name such as `spoke-us-west.kubeconfig` is not a valid cluster name, so CMF does not register the cluster. 2. Mount the Secret into the CMF pod with the `mountedVolumes` Helm values: ```yaml mountedVolumes: volumes: - name: kubeconfigs secret: secretName: cmf-kubeconfigs volumeMounts: - name: kubeconfigs mountPath: /etc/cmf/kubeconfig readOnly: true ``` The `mountPath` must equal the effective value of `cmf.k8s.kubeconfigDirectory`. If you override that property from its default `/etc/cmf/kubeconfig`, update `mountPath` to match. 3. Apply the updated Helm values with `helm upgrade`. Once the pod restarts with the mount in place, CMF detects the `kubeconfig` files automatically and begins registering the spoke clusters. To verify registration, follow [Register a cluster](#cmf-k8s-register-cluster). #### NOTE Any mechanism that materializes a Kubernetes Secret can back this mount — for example, [Vault Secrets Operator](https://developer.hashicorp.com/vault/docs/platform/k8s/vso) synchronizing from HashiCorp Vault. The `mountedVolumes` snippet above does not change. #### IMPORTANT Do not use a shared `ReadWriteMany` persistent volume as a drop-box for `kubeconfig` files. The Secret-based pattern keeps `kubeconfig` contents in the cluster’s Secret store — where Kubernetes RBAC applies and encryption-at-rest can be enabled for Secrets — and removes the need for a write path into the running CMF pod. For the `mountedVolumes` value schema and other Helm configuration, see [Install Confluent Manager for Apache Flink with Helm](../installation/helm.md#install-cmf-helm). For the kubeconfig directory setting, see the [configuration reference](#cmf-k8s-config-ref). ## Register a cluster CMF registers a cluster automatically when you place a kubeconfig file in the kubeconfig directory (`/etc/cmf/kubeconfig` by default, see [Configuration reference](#cmf-k8s-config-ref)). CMF derives the cluster name from the filename, creates a `KubernetesCluster` record in its database, and immediately begins [health monitoring](#cmf-k8s-heartbeat). CMF always registers the cluster where it is installed under the reserved name `default-k8s-cluster`. This cluster does not require a kubeconfig file. The name `default-k8s-cluster` is reserved and must not be used for any other cluster. #### NOTE Clusters cannot be manually created via the REST API. Registration is driven entirely by kubeconfig files in the watched directory. ## List and view clusters Use the `/cmf/api/v1/kubernetes-clusters` endpoint to view the state of all registered clusters. List all clusters: ```bash curl http://cmf:8080/cmf/api/v1/kubernetes-clusters ``` Get a specific cluster: ```bash curl http://cmf:8080/cmf/api/v1/kubernetes-clusters/my-cluster ``` Example response: ```json { "metadata": { "size": 2 }, "items": [ { "apiVersion": "cmf.confluent.io/v1", "kind": "KubernetesCluster", "metadata": { "name": "default-k8s-cluster", "creationTimestamp": "2025-01-10T08:00:00Z", "updateTimestamp": "2025-04-01T12:00:00Z" }, "spec": { "lifecycleState": "ACTIVE" }, "status": { "state": "CONNECTED", "lastHeartbeatTimestamp": "2025-04-10T10:55:00Z", "kubernetesVersion": "v1.29.2", "environments": [] } }, { "apiVersion": "cmf.confluent.io/v1", "kind": "KubernetesCluster", "metadata": { "name": "spoke-us-west", "creationTimestamp": "2025-03-01T09:00:00Z", "updateTimestamp": "2025-04-01T12:00:00Z" }, "spec": { "lifecycleState": "ACTIVE" }, "status": { "state": "CONNECTED", "lastHeartbeatTimestamp": "2025-04-10T10:55:00Z", "kubernetesVersion": "v1.30.1", "environments": ["my-env"] } } ] } ``` The response includes the following status fields: | Field | Description | |---------------------------------|--------------------------------------------------------------------------------------------------------------| | `status.state` | `CONNECTED`, `DISCONNECTED`, or `DECOMMISSIONED` | | `status.message` | Human-readable description of the current state or last error. Not set
while the cluster is `CONNECTED`. | | `status.lastHeartbeatTimestamp` | Timestamp of the last successful heartbeat | | `status.kubernetesVersion` | Kubernetes server version (only populated when `CONNECTED`) | | `status.environments` | Names of the environments bound to the cluster that you have permission
to view | ## Cluster states Each cluster is in one of three states: | State | Meaning | Write operations | Read operations | |------------------|--------------------------------|--------------------------|-------------------------| | `CONNECTED` | Cluster is reachable | Allowed | Live data | | `DISCONNECTED` | Cluster is unreachable | Most rejected (HTTP 503) | Stale data with warning | | `DECOMMISSIONED` | Cluster is permanently retired | Most rejected (HTTP 503) | Stale data with warning | When a cluster transitions to `DISCONNECTED`: - Mutating operations, such as create, update, delete, start, and suspend, on applications, statements, and savepoints in environments bound to that cluster return `HTTP 503` with a message identifying the cluster and its state. CMF also rejects creating or deleting an environment on the cluster and most writes to shared compute pools. CMF doesn’t reject deletes with `?force=true`, environment updates, writes to dedicated compute pools, or artifact and secret mapping operations. For compute pool details, see [Behavior on disconnected or decommissioned clusters](compute-pools.md#cmf-compute-pools-disconnected-cluster). - Read (GET and list) operations return the last known data, with a `status.warning` field indicating that data may be stale. When CMF’s heartbeat probe succeeds on a `DISCONNECTED` cluster, CMF transitions the cluster back to `CONNECTED` and resumes normal operation. CMF retries on each heartbeat cycle, so in most cases no manual intervention is required. Recovery is not automatic in the following cases: - The cluster’s kubeconfig file was removed. CMF probes only clusters that have a loaded kubeconfig, so the cluster stays `DISCONNECTED` until the file is restored. - The probe fails because the credentials in the loaded kubeconfig are no longer valid. The cluster stays `DISCONNECTED` until CMF loads an updated kubeconfig file for the cluster. For a Secret- or ConfigMap-mounted file, restart CMF after you update it. - Heartbeat monitoring is disabled with `cmf.k8s.heartbeatIntervalSeconds=0`. A `DISCONNECTED` or recommissioned cluster then does not return to `CONNECTED` on its own. Only adding or replacing its kubeconfig file triggers a new connectivity check. ## Decommission a cluster To permanently retire a cluster, set its `lifecycleState` to `DECOMMISSIONED`: ```bash curl -X PUT http://cmf:8080/cmf/api/v1/kubernetes-clusters/my-cluster \ -H "Content-Type: application/json" \ -d '{ "apiVersion": "cmf.confluent.io/v1", "kind": "KubernetesCluster", "metadata": { "name": "my-cluster" }, "spec": { "lifecycleState": "DECOMMISSIONED" } }' ``` A cluster cannot be decommissioned while environments are bound to it. Delete all environments bound to the cluster first. You cannot decommission `default-k8s-cluster`. Once decommissioned, the heartbeat stops monitoring the cluster. To recommission it, set `lifecycleState` back to `ACTIVE`. The cluster’s kubeconfig file must still be in the kubeconfig directory, or the cluster stays `DISCONNECTED`: ```bash curl -X PUT http://cmf:8080/cmf/api/v1/kubernetes-clusters/my-cluster \ -H "Content-Type: application/json" \ -d '{ "apiVersion": "cmf.confluent.io/v1", "kind": "KubernetesCluster", "metadata": { "name": "my-cluster" }, "spec": { "lifecycleState": "ACTIVE" } }' ``` #### NOTE To clear the labels or annotations on a `KubernetesCluster`, send an empty object, for example `labels: {}`, in the update request. Omitting the field, or sending `null`, preserves the existing values unchanged. A non-empty object replaces the existing set. To remove some labels, send only the labels you want to keep. ## Force-delete resources from a disconnected cluster When a cluster is `DISCONNECTED` or `DECOMMISSIONED`, normal deletion fails because CMF cannot reach the cluster. Use `?force=true` to remove a resource from CMF’s database. This may leave orphaned resources on the Kubernetes cluster that will need to be cleaned up manually when the cluster becomes reachable again: ```bash # Force-delete an application on a disconnected cluster curl -X DELETE \ "http://cmf:8080/cmf/api/v1/environments/my-env/applications/my-app?force=true" # Force-delete a compute pool curl -X DELETE \ "http://cmf:8080/cmf/api/v1/environments/my-env/compute-pools/my-pool?force=true" # Force-delete a SQL statement curl -X DELETE \ "http://cmf:8080/cmf/api/v1/environments/my-env/statements/my-statement?force=true" # Force-delete an environment (after all resources are removed) curl -X DELETE \ "http://cmf:8080/cmf/api/v1/environments/my-env?force=true" ``` ## Health monitoring (heartbeat) CMF probes every registered cluster on a fixed interval to detect connectivity changes: - **Interval**: every 60 seconds (configurable with `cmf.k8s.heartbeatIntervalSeconds`) - **Probe**: queries the cluster’s Kubernetes API server version - **Timeout**: 10 seconds per cluster (configurable with `cmf.k8s.heartbeatTimeoutSeconds`) - CMF skips `DECOMMISSIONED` clusters; the heartbeat does not run for them After a single failed or timed-out probe, the cluster transitions to `DISCONNECTED`. On the next successful probe, the cluster transitions back to `CONNECTED` and normal operation resumes automatically. Set `cmf.k8s.heartbeatIntervalSeconds=0` to disable heartbeat monitoring entirely. With heartbeat monitoring disabled, clusters do not recover from `DISCONNECTED` automatically. ## Configuration reference | Property | Default | Description | |------------------------------------|-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `cmf.k8s.heartbeatIntervalSeconds` | `60` | Interval in seconds between heartbeat rounds. Set to `0` to disable. | | `cmf.k8s.heartbeatTimeoutSeconds` | `10` | Per-cluster probe timeout in seconds. Must not exceed `heartbeatIntervalSeconds`. | | `cmf.k8s.kubeconfigDirectory` | `/etc/cmf/kubeconfig` | Directory watched for spoke cluster `kubeconfig` files. For a pattern
that mounts this directory from a Kubernetes Secret, see
[Mount kubeconfig files from a Kubernetes Secret](#cmf-k8s-kubeconfig-injection). | ## Related content - [Configure Environments in Confluent Manager for Apache Flink](environments.md#cmf-environments) - [Deploy and Manage Confluent Manager for Apache Flink Applications](../jobs/applications/overview.md#cmf-applications) - [Manage Compute Pools in Confluent Manager for Apache Flink](compute-pools.md#cmf-compute-pools) - [Configure Access Control for Confluent Manager for Apache Flink](access-control.md#cmf-access-control) - [REST APIs for Confluent Manager for Apache Flink](../clients-api/rest.md#af-rest-api)