Backup and restore using volume snapshots

Kubernetes volume snapshots provide a standardized way to create copies of persistent volumes at a point in time without creating new volumes. Backing up your directory data with volume snapshots lets you perform rapid recovery from the last snapshot point. Volume snapshot backups also facilitate testing by letting you initialize DS with sample data.

In the CDM, the DS data, change log, and configuration are stored in the same persistent volume. This ensures the volume snapshot captures DS data and changelog together.

The backup and restore procedure using volume snapshots described here is meant for use in ForgeOps release 7.4 deployment environments where ds-operator is not used.

Backup

Set up backup

The Kustomize overlays necessary to set up volume snapshots of the CDM deployed to the prod namespace are provided in the kustomize/overlay/ds-snapshot directory of the forgeops repository. These overlays are not handled by the forgeops command.

When enabled, the default setup of volume snapshot takes snapshot of the data-ds-idrepo-0 and data-ds-cts-0 PVCs once a day.

To enable volume snapshot of DS data from the my-namespace namespace using the default settings, perform the following steps:

In a terminal window, change to the ds-snapshot subdirectory under the kustomize/overlay directory:
```
$ cd /path/to/forgeops/kustomize/overlay/ds-snapshot
```
Copy the content of the prod directory to a new directory with the name of the namespace where you have deployed CDM:
```
$ cp -rp ./prod ./my-namespace
```
Change to the my-namespace directory.
Edit the /rbac/namespace.yaml file and change the last line to specify the namespace in which CDM has been deployed.

Set up the configuration map and enable volume snapshot backup using the kubectl apply command:

$ kubectl apply --kustomize configmap --namespace my-namespace
$ kubectl apply --kustomize rbac --namespace my-namespace
$ kubectl apply --kustomize idrepo --namespace my-namespace

Optionally, if you want to back up the cts as well, then run the following:
```
$ kubectl apply --kustomize cts --namespace my-namespace
```

View the volume snapshots that are available for restore, using this command:

$ kubectl get volumesnapshots --namespace my-namespace

NAME                               READYTOUSE   SOURCEPVC          SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT
          CREATIONTIME   AGE
ds-idrepo-snapshot-20231117-1320   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-be3f4a44-cfb2-4f68-aa2b-60902
bb44192   3h29m          3h29m
ds-idrepo-snapshot-20231117-1330   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-7bcf6779-382d-40e3-9c9f-edf31
c54768e   3h19m          3h19m
ds-idrepo-snapshot-20231117-1340   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-c9c88332-ad05-4880-bda7-48616
ec13579   3h9m           3h9m
ds-idrepo-snapshot-20231117-1401   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-1f3f4ce9-0083-447f-9803-f6b45
e03ac27   167m           167m
ds-idrepo-snapshot-20231117-1412   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-4c39c095-0891-4da8-ae61-fac78
c7147ff   156m           156m

Customize backup schedule

When enabled, volume snapshots are created once every day by default, and purged after three days. To modify the default schedule and purge delay, edit the schedule.yaml file in cts and idrepo directories, and run the kubectl apply command.

Examples for scheduling snapshots

To schedule snapshot twice a day, at 12:00 noon and midnight:
```
...
  spec:
    schedule: "0 0/12 * * *"
...
```

To schedule snapshot every 8 hours:

...
  spec:
    schedule: "0 */8 * * *"
...

Examples for purging schedule

To schedule purge after 4 days:

...
         env:
           - name: PURGE_DELAY
             value: "-4 day"

To schedule purge after a week:

...
         env:
           - name: PURGE_DELAY
             value: "-7 day"

Restore from volume snapshot

ForgeOps team provides the snapshot-restore.sh script to restore the DS instances in the CDM. This script restores a DS instance from the latest available snapshot, by default.

The snapshot-restore.sh script requires the JQ utility to manage JSON files used in restore operations. You must install JQ before using the snapshot-restore.sh script.

There are two options when using the snapshot-restore.sh script to restore a DS from a volume snapshot:

Full: Use the full option to fully restore a DS instance from a volume snapshot. In this option, the DS is scaled down to 0 pods before restoring data. The data is restored to an existing PVC from a snapshot. This operation requires downtime.
Selective: Use the selective option to restore a select portion of DS data from volume snapshot. The selective restore creates a new temporary DS instance with a new DS pod. You can selectively export from the temporary DS pod and import into your functional DS instance. After restoring data, you can clean up the temporary resources.

The snapshot-restore.sh command is available in the bin directory of the forgeops repository. To learn more about the snapshot-restore.sh command, use snapshot-restore.sh --help command to learn more about the command and its options.

Restore examples

Trial run without actually restoring DS data

In a terminal window, change to /path/to/forgeops/bin directory.
Set your Kubernetes context to the correct cluster and namespace.

Run the snapshot-restore.sh command with --dryrun option:

$ ./snapshot-restore.sh --dryrun --namespace my-namespace full idrepo

./snapshot-restore.sh --dryrun --namespace my-namespace full idrepo
/usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/sts-restore.json -n my-namespace
/usr/local/bin/kubectl delete pvc data-ds-idrepo-0 -n my-namespace
/usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/data-ds-idrepo-0.json -n my-namespace
/usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/sts.json -n my-namespace

Full restore of the idrepo instance from the latest available volume snapshot

In a terminal window, change to /path/to/forgeops/bin directory.
Set your Kubernetes context to the correct cluster and namespace.

Get a list of available volume snapshots:

$ kubectl get volumesnapshots --namespace my-namespace

Restore full DS instance:

$ ./snapshot-restore.sh --namespace my-namespace full idrepo

Verify that DS data has been restored.

Selective restore from a specific volume snapshot and storing data in a user-defined storage path

In a terminal window, change to /path/to/forgeops/bin directory.
Set your Kubernetes context to the correct cluster and namespace.

View the available volume snapshots, using this command:

$ kubectl get volumesnapshots --namespace my-namespace

Perform selective restore trial run:

$ ./snapshot-restore.sh --dryrun --path /tmp/ds-restore --snapshot ds-idrepo-snapshot-20231121-2250 --namespace my-namespace selective idrepo

VolumeSnapshot ds-idrepo-snapshot-20231121-2250 is ready to use
/usr/local/bin/kubectl apply -f /tmp/ds-rest/sts-restore.json -n my-namespace
/usr/local/bin/kubectl apply -f /tmp/ds-rest/svc.json -n my-namespace

Perform selective restore using a specific snapshot:

$ ./snapshot-restore.sh --path /tmp/ds-restore --snapshot ds-idrepo-snapshot-20231121-2250 --namespace my-namespace selective idrepo

statefulset.apps/ds-idrepo-restore created
service/ds-idrepo configured

Verify a new ds-idrepo-restore-0 pod is created:

$ kubectl get pods
NAME                          READY   STATUS      RESTARTS   AGE
admin-ui-656db67f54-2brbf     1/1     Running     0          3h17m
am-7fffff59fd-mkks5           1/1     Running     0          107m
amster-hgkv9                  0/1     Completed   0          3h18m
ds-idrepo-0                   1/1     Running     0          39m
ds-idrepo-restore-0           1/1     Running     0          2m40s
end-user-ui-df49f79d4-n4q54   1/1     Running     0          3h17m
idm-fc88578bf-lqcdj           1/1     Running     0          3h18m
login-ui-5945d48fc6-ljxw2     1/1     Running     0          3h17m

The ds-idrepo-restore-0 pod is temporary and not to be used as a complete DS instance. You can export required data from the temporary pod, and import data into your functional DS instance.

The following sample commands are meant to be examples and are not to be used in production.

Connect to the ds-idrepo-restore-0 pod and run the export-ldif command, for example:

$ kubectl exec ds-idrepo-restore-0 -it — bash
$ export-ldif \
 --includeBranch dc=example,dc=com \
 --backendId userData \
 --ldifFile /path/to/DS/ldif/my-export.ldif \
 --offline

Copy the exported LDIF file from ds-idrepo-restore-0 pod to a local folder:

$ kubectl cp ds-idrepo-restore-0:/path/to/DS/ldif/my-export.ldif /path/to/local/destination

Copy the exported file from the local folder to the ds-idrepo-0 pod:

$ kubectl cp /path/to/local/destination/my-export.ldif ds-idrepo-0:/path/to/DS/ldif

Import data into the ds-idrepo instance:

$ kubectl exec ds-idrepo-0 -it — bash
$ import-ldif --includeBranch dc=example,dc=com --backendId userData --ldifFile ds-idrepo-0:/path/to/DS/ldif/my-export.ldif

Clean up resources from selective restore:

$ ./snapshot-restore.sh clean idrepo

statefulset.apps "ds-idrepo-restore" deleted
persistentvolumeclaim "data-ds-idrepo-restore-0" deleted