Back Up and Restore SQLite Metadata of Confluent Manager for Apache Flink with Volume Snapshots

This topic describes how to back up and restore the SQLite metadata of Confluent Manager for Apache Flink® (CMF) with Kubernetes volume snapshots.

When CMF uses the default SQLite database, its metadata is on the persistent volume claim (PVC) confluent-manager-for-apache-flink-pvc, or <name>-pvc if you set the nameOverride Helm value to <name>. You back it up and restore it with the Kubernetes Volume Snapshots feature, which Google Cloud, AWS, and Azure support. A volume snapshot is a Kubernetes resource that captures the state of a PVC at a specific point in time. To learn more about persistent volumes, see Kubernetes Persistent Volumes.

A restore brings back the metadata as it was when you took the snapshot, so take snapshots regularly. With some cloud providers, such as AWS, you can restore a snapshot only in the region where you took it. To restore it in another region, copy it to that region first, as your cloud provider describes. The copy has its own snapshot handle.

Prerequisites

Note the following common terminology:

Custom Resource Definition (CRD): A component that extends the Kubernetes API.
Container Storage Interface (CSI): An industry standard that enables storage systems to seamlessly integrate with container orchestration platforms like Kubernetes.

The following prerequisites depend on your cloud provider. Complete them on every cluster where you create or restore snapshots, including a replacement cluster.

AWS

Complete the following steps to enable Volume Snapshots on Amazon Elastic Kubernetes Service (EKS):

Enable the EKS add-on for the Amazon Elastic Block Store (EBS) Container Storage Interface (CSI) driver.
Configure an Amazon Identity and Access Management (IAM) role for the driver’s controller. Attach the AmazonEBSCSIDriverPolicy managed policy to grant the necessary permissions, such as ec2:CreateSnapshot.

Use the following commands to install the Container Storage Interface (CSI) snapshot controller components and custom resource definition (CRD):

# CRDs
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
# Controllers
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

Create the storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

Before installing CMF, ensure that the default storage class is set to ebs-sc.

# Disable gp2 as the default storage class
kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
# Enable ebs-sc as the default storage class
kubectl patch storageclass ebs-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Google Cloud

The Google Cloud Persistent Disk CSI driver is a managed add-on and is enabled by default for Autopilot clusters. You might need to enable it for standard clusters. For more information, see Using the Compute Engine persistent disk CSI Driver.
You do not need to install CRDs for volume snapshots. They are pre-installed.

OpenShift on Azure

OpenShift provides a snapshot controller operator (csi-snapshot-controller-operator), so you don’t need to install CRDs.

Create a snapshot

The following steps assume that CMF is installed in the confluent namespace.

Install the VolumeSnapshotClass.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: cmf-snapshot
driver: <ebs.csi.aws.com/pd.csi.storage.gke.io/disk.csi.azure.com>
deletionPolicy: Retain

With deletionPolicy: Retain, the snapshot stays with your cloud provider when its VolumeSnapshot is deleted, for example together with its namespace.

Create the snapshot.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: first-snapshot
  namespace: confluent
spec:
  volumeSnapshotClassName: cmf-snapshot
  source:
    # The specified PVC should point to CMF PVC.
    persistentVolumeClaimName: confluent-manager-for-apache-flink-pvc

Wait until the snapshot is ready to use:

kubectl wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/first-snapshot -n confluent --timeout=10m

Record the snapshot handle, which you need to restore the snapshot on another cluster:

kubectl get volumesnapshotcontent \
  "$(kubectl get volumesnapshot first-snapshot -n confluent -o jsonpath='{.status.boundVolumeSnapshotContentName}')" \
  -o jsonpath='{.status.snapshotHandle}'

You can also find the snapshot handle in your cloud provider’s UI, such as the AWS Console.

Restore from a snapshot

Follow these steps to install CMF with its metadata restored from a volume snapshot. The Helm chart creates the CMF PVC from the snapshot, so the PVC must not exist yet. If CMF is still installed, for example because you restore on the same cluster, uninstall it first with helm uninstall cmf -n confluent. This deletes the PVC that holds the current database, but not the VolumeSnapshot. If the VolumeSnapshot still exists in the namespace, skip the steps that create it, and use its name in the Helm values.

On a replacement cluster, complete the prerequisites for your cloud provider, create the confluent namespace, and install the VolumeSnapshotClass, as described in Create a snapshot.

Create the VolumeSnapshotContent resource that references the snapshot handle.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: restored-vsc
spec:
  deletionPolicy: Retain
  driver: <ebs.csi.aws.com/pd.csi.storage.gke.io/disk.csi.azure.com>
  source:
    snapshotHandle: <snapshot_handle>
  volumeSnapshotClassName: cmf-snapshot
  volumeSnapshotRef:
    name: restored-snapshot
    namespace: confluent

Create a VolumeSnapshot resource that references the VolumeSnapshotContent, and wait until it is ready to use.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: restored-snapshot
  namespace: confluent
spec:
  volumeSnapshotClassName: cmf-snapshot
  source:
    volumeSnapshotContentName: restored-vsc

kubectl wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/restored-snapshot -n confluent --timeout=10m

Create a YAML file with the following Helm values, so that the Helm chart creates the CMF PVC from the snapshot:
```
# restore-cmf.yaml
pvcOverrides:
  spec:
    dataSource:
      name: restored-snapshot
      kind: VolumeSnapshot
      apiGroup: snapshot.storage.k8s.io
```
The pvcOverrides value requires the Helm chart of CMF 2.5.0 or later. Earlier charts ignore it, and CMF then starts with an empty database. The PVC must be at least as large as the snapshot, so keep persistence.dataVolumeCapacity at the value you used before. Keep these values unchanged in the files that you use for later helm upgrade commands, because Kubernetes does not allow the data source of an existing PVC to change.

Install CMF with Helm, together with the rest of your Helm values:

helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink -n confluent -f values.yaml -f restore-cmf.yaml