# Back Up and Restore SQLite Metadata of Confluent Manager for Apache Flink with Volume Snapshots This topic describes how to back up and restore the SQLite metadata of Confluent Manager for Apache Flink® (CMF) with Kubernetes volume snapshots. When CMF uses the default SQLite database, its metadata is on the persistent volume claim (PVC) `confluent-manager-for-apache-flink-pvc`, or `-pvc` if you set the `nameOverride` Helm value to ``. You back it up and restore it with the [Kubernetes Volume Snapshots feature](https://kubernetes.io/docs/concepts/storage/volume-snapshots/), which Google Cloud, AWS, and Azure support. A volume snapshot is a Kubernetes resource that captures the state of a PVC at a specific point in time. To learn more about persistent volumes, see [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). A restore brings back the metadata as it was when you took the snapshot, so take snapshots regularly. With some cloud providers, such as AWS, you can restore a snapshot only in the region where you took it. To restore it in another region, copy it to that region first, as your cloud provider describes. The copy has its own snapshot handle. ## Prerequisites Note the following common terminology: - Custom Resource Definition (CRD): A component that extends the Kubernetes API. - Container Storage Interface (CSI): An industry standard that enables storage systems to seamlessly integrate with container orchestration platforms like Kubernetes. The following prerequisites depend on your cloud provider. Complete them on every cluster where you create or restore snapshots, including a replacement cluster. ### AWS Complete the following steps to enable Volume Snapshots on Amazon Elastic Kubernetes Service (EKS): 1. Enable the EKS add-on for the Amazon Elastic Block Store (EBS) Container Storage Interface (CSI) driver. 2. Configure an Amazon Identity and Access Management (IAM) role for the driver’s controller. Attach the `AmazonEBSCSIDriverPolicy` managed policy to grant the necessary permissions, such as `ec2:CreateSnapshot`. 3. Use the following commands to install the Container Storage Interface (CSI) snapshot controller components and custom resource definition (CRD): ```bash # CRDs kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml # Controllers kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml ``` 4. Create the storage class: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ebs-sc provisioner: ebs.csi.aws.com parameters: type: gp3 reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer ``` 5. Before installing CMF, ensure that the default storage class is set to `ebs-sc`. ```bash # Disable gp2 as the default storage class kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' # Enable ebs-sc as the default storage class kubectl patch storageclass ebs-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' ``` ### Google Cloud * The Google Cloud Persistent Disk CSI driver is a managed add-on and is enabled by default for Autopilot clusters. You might need to enable it for standard clusters. For more information, see [Using the Compute Engine persistent disk CSI Driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver). * You do not need to install CRDs for volume snapshots. They are pre-installed. ### OpenShift on Azure * OpenShift provides a snapshot controller operator (`csi-snapshot-controller-operator`), so you don’t need to install CRDs. ## Create a snapshot The following steps assume that CMF is installed in the `confluent` namespace. 1. Install the `VolumeSnapshotClass`. ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: cmf-snapshot driver: deletionPolicy: Retain ``` With `deletionPolicy: Retain`, the snapshot stays with your cloud provider when its `VolumeSnapshot` is deleted, for example together with its namespace. 2. Create the snapshot. ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: first-snapshot namespace: confluent spec: volumeSnapshotClassName: cmf-snapshot source: # The specified PVC should point to CMF PVC. persistentVolumeClaimName: confluent-manager-for-apache-flink-pvc ``` 3. Wait until the snapshot is ready to use: ```bash kubectl wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/first-snapshot -n confluent --timeout=10m ``` 4. Record the snapshot handle, which you need to restore the snapshot on another cluster: ```bash kubectl get volumesnapshotcontent \ "$(kubectl get volumesnapshot first-snapshot -n confluent -o jsonpath='{.status.boundVolumeSnapshotContentName}')" \ -o jsonpath='{.status.snapshotHandle}' ``` You can also find the snapshot handle in your cloud provider’s UI, such as the AWS Console. ## Restore from a snapshot Follow these steps to install CMF with its metadata restored from a volume snapshot. The Helm chart creates the CMF PVC from the snapshot, so the PVC must not exist yet. If CMF is still installed, for example because you restore on the same cluster, uninstall it first with `helm uninstall cmf -n confluent`. This deletes the PVC that holds the current database, but not the `VolumeSnapshot`. If the `VolumeSnapshot` still exists in the namespace, skip the steps that create it, and use its name in the Helm values. 1. On a replacement cluster, complete the prerequisites for your cloud provider, create the `confluent` namespace, and install the `VolumeSnapshotClass`, as described in [Create a snapshot](#cmf-dr-create-snapshot). 2. Create the `VolumeSnapshotContent` resource that references the snapshot handle. ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: name: restored-vsc spec: deletionPolicy: Retain driver: source: snapshotHandle: volumeSnapshotClassName: cmf-snapshot volumeSnapshotRef: name: restored-snapshot namespace: confluent ``` 3. Create a `VolumeSnapshot` resource that references the `VolumeSnapshotContent`, and wait until it is ready to use. ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: restored-snapshot namespace: confluent spec: volumeSnapshotClassName: cmf-snapshot source: volumeSnapshotContentName: restored-vsc ``` ```bash kubectl wait --for=jsonpath='{.status.readyToUse}'=true volumesnapshot/restored-snapshot -n confluent --timeout=10m ``` 4. Create a YAML file with the following Helm values, so that the Helm chart creates the CMF PVC from the snapshot: ```yaml # restore-cmf.yaml pvcOverrides: spec: dataSource: name: restored-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io ``` The `pvcOverrides` value requires the Helm chart of CMF 2.5.0 or later. Earlier charts ignore it, and CMF then starts with an empty database. The PVC must be at least as large as the snapshot, so keep `persistence.dataVolumeCapacity` at the value you used before. Keep these values unchanged in the files that you use for later `helm upgrade` commands, because Kubernetes does not allow the data source of an existing PVC to change. 5. Install CMF with Helm, together with the rest of your Helm values: ```bash helm upgrade --install cmf confluentinc/confluent-manager-for-apache-flink -n confluent -f values.yaml -f restore-cmf.yaml ``` ## Related content - [Disaster Recovery for Confluent Manager for Apache Flink](overview.md#backup-restore-cmf) - [Configure Storage for Confluent Manager for Apache Flink](../installation/storage.md#cmf-storage) - [Install Confluent Manager for Apache Flink with Helm](../installation/helm.md#install-cmf-helm)