Manage Multiple Kubernetes Clusters in Confluent Manager for Apache Flink

Confluent Manager for Apache Flink® (CMF) supports managing Flink workloads across multiple Kubernetes clusters. A single CMF instance can register and manage additional clusters, deploy environments and applications on them, and monitor their health, all through the same REST API.

CMF represents every Kubernetes cluster it manages as a KubernetesCluster resource, including the cluster where CMF itself is installed (the hub cluster). By default, CMF deploys Flink resources on this hub cluster. Multi-cluster support allows additional spoke clusters to be registered. Once registered, environments can be bound to any cluster, and Flink applications, compute pools, and SQL statements can be deployed on that cluster.

This topic describes how to register, monitor, and decommission Kubernetes clusters in CMF.

Note

You can view, decommission, and recommission Kubernetes clusters with the REST API or the CMF UI. Clusters are registered only through kubeconfig files.

Prerequisites

Before registering a spoke cluster:

1. Install the Flink Kubernetes Operator (FKO) on the spoke cluster. The FKO and its Custom Resource Definitions (CRDs) (FlinkDeployment, FlinkSessionJob, FlinkStateSnapshot) must be present before the cluster is added to CMF. Registering a cluster before FKO is installed leads to an inconsistent state where the cluster appears connected but workload operations are not monitored correctly. To recover, install FKO on the cluster and then decommission and recommission the cluster via the API.

   CMF validates the FKO installation when a cluster is first registered and again whenever it reconnects after a DISCONNECTED transition. This validation is advisory — CMF logs a warning but does not block registration if the check fails.

2. Place a kubeconfig file for the cluster in CMF‘s kubeconfig directory (/etc/cmf/kubeconfig by default, configurable with cmf.k8s.kubeconfigDirectory). The filename must exactly match the cluster name used during registration. For example, a file named my-cluster registers the cluster under the name my-cluster. The filename must be a valid cluster name: lowercase letters, digits, and hyphens, starting with a letter, ending with a letter or digit, and at most 45 characters long. CMF does not register a file with any other name. If CMF runs in a Kubernetes pod, see Mount kubeconfig files from a Kubernetes Secret for a Secret-backed mounting pattern.

3. Ensure network and RBAC access. CMF must be able to reach the spoke cluster’s API server, and the kubeconfig must grant CMF sufficient permissions to connect to the API server and monitor Flink resources on the cluster.

Kubeconfig management

CMF watches the kubeconfig directory for file changes at runtime, so clusters can be registered and deregistered without restarting CMF. Adding a kubeconfig file registers the cluster, and replacing a file, such as for credential rotation, re-initializes the connection to that cluster automatically. If the kubeconfig files are mounted from a Kubernetes Secret or ConfigMap, CMF does not detect changes to the contents of an existing file, so restart CMF after you update the Secret or ConfigMap. Removing the file of an ACTIVE cluster does not deregister it. The cluster stays registered and becomes DISCONNECTED. To deregister a cluster, decommission it first, and then remove its kubeconfig file. CMF then deletes the cluster record.

Note

The maximum kubeconfig file size is 10 MB.

For details on staging kubeconfig files in a containerized deployment, see Mount kubeconfig files from a Kubernetes Secret. For other kubeconfig settings, see the configuration reference.

Mount kubeconfig files from a Kubernetes Secret

Because Confluent Manager for Apache Flink (CMF) runs in a container, you cannot durably place kubeconfig files on the pod filesystem by copying them in at runtime — files written with tools like kubectl cp are lost on pod restart, and staging files on a shared ReadWriteMany persistent volume so something outside the pod can write to it is discouraged. The recommended pattern is to store kubeconfig files in a Kubernetes Secret and volume-mount the Secret at cmf.k8s.kubeconfigDirectory.

Each key in the Secret becomes a file inside the mounted directory, and CMF registers each file as a spoke cluster under the filename. The filename-as-cluster-name rule from the Prerequisites applies here — choose Secret keys that match the cluster names you intend to register.

Note

Although CMF supports kubeconfig files up to 10 MB, Kubernetes Secret objects are typically limited to about 1 MiB total size per object. If a kubeconfig is too large to fit in a Secret, reduce its size where possible (for example, avoid embedding large certificate or key data), or use an alternative delivery mechanism that places the kubeconfig file in cmf.k8s.kubeconfigDirectory without storing it in a Kubernetes Secret.

1. Create a Kubernetes Secret containing the kubeconfig files:

   kubectl create secret generic cmf-kubeconfigs \
     --from-file=spoke-us-west=/local/path/to/us-west.kubeconfig \
     --from-file=spoke-eu-central=/local/path/to/eu-central.kubeconfig \
     -n <your-cmf-namespace>
   

   The key on the left of each = is the filename that appears in the mounted directory and the cluster name CMF registers. The path on the right is the source kubeconfig on your workstation. Do not append an extension such as .kubeconfig to the left-hand side. A name such as spoke-us-west.kubeconfig is not a valid cluster name, so CMF does not register the cluster.

2. Mount the Secret into the CMF pod with the mountedVolumes Helm values:

   mountedVolumes:
     volumes:
       - name: kubeconfigs
         secret:
           secretName: cmf-kubeconfigs
     volumeMounts:
       - name: kubeconfigs
         mountPath: /etc/cmf/kubeconfig
         readOnly: true
   

   The mountPath must equal the effective value of cmf.k8s.kubeconfigDirectory. If you override that property from its default /etc/cmf/kubeconfig, update mountPath to match.

3. Apply the updated Helm values with helm upgrade. Once the pod restarts with the mount in place, CMF detects the kubeconfig files automatically and begins registering the spoke clusters. To verify registration, follow Register a cluster.

Note

Any mechanism that materializes a Kubernetes Secret can back this mount — for example, Vault Secrets Operator synchronizing from HashiCorp Vault. The mountedVolumes snippet above does not change.

Important

Do not use a shared ReadWriteMany persistent volume as a drop-box for kubeconfig files. The Secret-based pattern keeps kubeconfig contents in the cluster’s Secret store — where Kubernetes RBAC applies and encryption-at-rest can be enabled for Secrets — and removes the need for a write path into the running CMF pod.

For the mountedVolumes value schema and other Helm configuration, see Install Confluent Manager for Apache Flink with Helm. For the kubeconfig directory setting, see the configuration reference.

Register a cluster

CMF registers a cluster automatically when you place a kubeconfig file in the kubeconfig directory (/etc/cmf/kubeconfig by default, see Configuration reference). CMF derives the cluster name from the filename, creates a KubernetesCluster record in its database, and immediately begins health monitoring.

CMF always registers the cluster where it is installed under the reserved name default-k8s-cluster. This cluster does not require a kubeconfig file. The name default-k8s-cluster is reserved and must not be used for any other cluster.

Note

Clusters cannot be manually created via the REST API. Registration is driven entirely by kubeconfig files in the watched directory.

List and view clusters

Use the /cmf/api/v1/kubernetes-clusters endpoint to view the state of all registered clusters.

List all clusters:

curl http://cmf:8080/cmf/api/v1/kubernetes-clusters

Get a specific cluster:

curl http://cmf:8080/cmf/api/v1/kubernetes-clusters/my-cluster

Example response:

{
  "metadata": {
    "size": 2
  },
  "items": [
    {
      "apiVersion": "cmf.confluent.io/v1",
      "kind": "KubernetesCluster",
      "metadata": {
        "name": "default-k8s-cluster",
        "creationTimestamp": "2025-01-10T08:00:00Z",
        "updateTimestamp": "2025-04-01T12:00:00Z"
      },
      "spec": {
        "lifecycleState": "ACTIVE"
      },
      "status": {
        "state": "CONNECTED",
        "lastHeartbeatTimestamp": "2025-04-10T10:55:00Z",
        "kubernetesVersion": "v1.29.2",
        "environments": []
      }
    },
    {
      "apiVersion": "cmf.confluent.io/v1",
      "kind": "KubernetesCluster",
      "metadata": {
        "name": "spoke-us-west",
        "creationTimestamp": "2025-03-01T09:00:00Z",
        "updateTimestamp": "2025-04-01T12:00:00Z"
      },
      "spec": {
        "lifecycleState": "ACTIVE"
      },
      "status": {
        "state": "CONNECTED",
        "lastHeartbeatTimestamp": "2025-04-10T10:55:00Z",
        "kubernetesVersion": "v1.30.1",
        "environments": ["my-env"]
      }
    }
  ]
}

The response includes the following status fields:

Field	Description
status.state	CONNECTED, DISCONNECTED, or DECOMMISSIONED
status.message	Human-readable description of the current state or last error. Not set while the cluster is CONNECTED.
status.lastHeartbeatTimestamp	Timestamp of the last successful heartbeat
status.kubernetesVersion	Kubernetes server version (only populated when CONNECTED)
status.environments	Names of the environments bound to the cluster that you have permission to view

Cluster states

Each cluster is in one of three states:

State	Meaning	Write operations	Read operations
CONNECTED	Cluster is reachable	Allowed	Live data
DISCONNECTED	Cluster is unreachable	Most rejected (HTTP 503)	Stale data with warning
DECOMMISSIONED	Cluster is permanently retired	Most rejected (HTTP 503)	Stale data with warning

When a cluster transitions to DISCONNECTED:

• Mutating operations, such as create, update, delete, start, and suspend, on applications, statements, and savepoints in environments bound to that cluster return HTTP 503 with a message identifying the cluster and its state. CMF also rejects creating or deleting an environment on the cluster and most writes to shared compute pools. CMF doesn’t reject deletes with ?force=true, environment updates, writes to dedicated compute pools, or artifact and secret mapping operations. For compute pool details, see Behavior on disconnected or decommissioned clusters.

• Read (GET and list) operations return the last known data, with a status.warning field indicating that data may be stale.

When CMF’s heartbeat probe succeeds on a DISCONNECTED cluster, CMF transitions the cluster back to CONNECTED and resumes normal operation. CMF retries on each heartbeat cycle, so in most cases no manual intervention is required. Recovery is not automatic in the following cases:

• The cluster’s kubeconfig file was removed. CMF probes only clusters that have a loaded kubeconfig, so the cluster stays DISCONNECTED until the file is restored.

• The probe fails because the credentials in the loaded kubeconfig are no longer valid. The cluster stays DISCONNECTED until CMF loads an updated kubeconfig file for the cluster. For a Secret- or ConfigMap-mounted file, restart CMF after you update it.

• Heartbeat monitoring is disabled with cmf.k8s.heartbeatIntervalSeconds=0. A DISCONNECTED or recommissioned cluster then does not return to CONNECTED on its own. Only adding or replacing its kubeconfig file triggers a new connectivity check.

Decommission a cluster

To permanently retire a cluster, set its lifecycleState to DECOMMISSIONED:

curl -X PUT http://cmf:8080/cmf/api/v1/kubernetes-clusters/my-cluster \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "cmf.confluent.io/v1",
    "kind": "KubernetesCluster",
    "metadata": { "name": "my-cluster" },
    "spec": { "lifecycleState": "DECOMMISSIONED" }
  }'

A cluster cannot be decommissioned while environments are bound to it. Delete all environments bound to the cluster first. You cannot decommission default-k8s-cluster.

Once decommissioned, the heartbeat stops monitoring the cluster. To recommission it, set lifecycleState back to ACTIVE. The cluster’s kubeconfig file must still be in the kubeconfig directory, or the cluster stays DISCONNECTED:

curl -X PUT http://cmf:8080/cmf/api/v1/kubernetes-clusters/my-cluster \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "cmf.confluent.io/v1",
    "kind": "KubernetesCluster",
    "metadata": { "name": "my-cluster" },
    "spec": { "lifecycleState": "ACTIVE" }
  }'

Note

To clear the labels or annotations on a KubernetesCluster, send an empty object, for example labels: {}, in the update request. Omitting the field, or sending null, preserves the existing values unchanged. A non-empty object replaces the existing set. To remove some labels, send only the labels you want to keep.

Force-delete resources from a disconnected cluster

When a cluster is DISCONNECTED or DECOMMISSIONED, normal deletion fails because CMF cannot reach the cluster. Use ?force=true to remove a resource from CMF’s database. This may leave orphaned resources on the Kubernetes cluster that will need to be cleaned up manually when the cluster becomes reachable again:

# Force-delete an application on a disconnected cluster
curl -X DELETE \
  "http://cmf:8080/cmf/api/v1/environments/my-env/applications/my-app?force=true"

# Force-delete a compute pool
curl -X DELETE \
  "http://cmf:8080/cmf/api/v1/environments/my-env/compute-pools/my-pool?force=true"

# Force-delete a SQL statement
curl -X DELETE \
  "http://cmf:8080/cmf/api/v1/environments/my-env/statements/my-statement?force=true"

# Force-delete an environment (after all resources are removed)
curl -X DELETE \
  "http://cmf:8080/cmf/api/v1/environments/my-env?force=true"

Health monitoring (heartbeat)

CMF probes every registered cluster on a fixed interval to detect connectivity changes:

• Interval: every 60 seconds (configurable with cmf.k8s.heartbeatIntervalSeconds)

• Probe: queries the cluster’s Kubernetes API server version

• Timeout: 10 seconds per cluster (configurable with cmf.k8s.heartbeatTimeoutSeconds)

• CMF skips DECOMMISSIONED clusters; the heartbeat does not run for them

After a single failed or timed-out probe, the cluster transitions to DISCONNECTED. On the next successful probe, the cluster transitions back to CONNECTED and normal operation resumes automatically.

Set cmf.k8s.heartbeatIntervalSeconds=0 to disable heartbeat monitoring entirely. With heartbeat monitoring disabled, clusters do not recover from DISCONNECTED automatically.

Configuration reference

Property	Default	Description
cmf.k8s.heartbeatIntervalSeconds	60	Interval in seconds between heartbeat rounds. Set to 0 to disable.
cmf.k8s.heartbeatTimeoutSeconds	10	Per-cluster probe timeout in seconds. Must not exceed heartbeatIntervalSeconds.
cmf.k8s.kubeconfigDirectory	/etc/cmf/kubeconfig	Directory watched for spoke cluster kubeconfig files. For a pattern that mounts this directory from a Kubernetes Secret, see Mount kubeconfig files from a Kubernetes Secret.

Related content

• Configure Environments in Confluent Manager for Apache Flink

• Deploy and Manage Confluent Manager for Apache Flink Applications

• Manage Compute Pools in Confluent Manager for Apache Flink

• Configure Access Control for Confluent Manager for Apache Flink

• REST APIs for Confluent Manager for Apache Flink