Backup and restore using volume snapshots

Kubernetes volume snapshots provide a standardized way to create copies of persistent volumes at a point in time without creating new volumes. Backing up your directory data with volume snapshots lets you perform rapid recovery from the last snapshot point. Volume snapshot backups also facilitate testing by letting you initialize DS with sample data.

In ForgeOps deployments, the DS data, changelog, and configuration are stored in the same persistent volume. This ensures the volume snapshot captures DS data and changelog together.

The backup and restore procedures using volume snapshots described here are meant for use with ForgeOps deployments where the DS operator is not used.

Backup

Set up backup

Kustomize overlays and Helm values necessary for configuring volume snapshots are already provided, but they have not been enabled to take backup. The default volume snapshot setup takes snapshots of the data-ds-idrepo-0 and data-ds-cts-0 PVCs once a day.

To enable volume snapshots of DS data from the my-namespace namespace using the default settings, perform the following steps:

In a Kustomize-based deployment
In a Helm-based deployment

The default Kustomize overlays are provided in the kustomize/overlay/ds-snapshot directory of the forgeops repository.

In a terminal window, change to the ds-snapshot subdirectory under the kustomize/overlay directory:
```
$ cd /path/to/forgeops/kustomize/overlay/ds-snapshot
```
Copy the content of the prod directory to a new directory with the name of the namespace where you have performed a ForgeOps deployment:
```
$ cp -rp ./prod ./my-namespace
```
Change to the my-namespace directory.
Edit the rbac/namespace.yaml file and change the last line to specify the namespace where you performed the ForgeOps deployment.

Set up the configuration map and enable volume snapshot backup using the kubectl apply commands:

$ kubectl apply --kustomize configmap --namespace my-namespace
$ kubectl apply --kustomize rbac --namespace my-namespace
$ kubectl apply --kustomize idrepo --namespace my-namespace

(Optional) If you want to back up the cts as well, then run the following:
```
$ kubectl apply --kustomize cts --namespace my-namespace
```

The default Helm values are provided in the charts/identity-platform/values.yaml file of the forgeops repository.

In a terminal window, change to the charts/identity-platform directory:
```
$ cd /path/to/forgeops/charts/identity-platform
```
Copy the values.yaml file, so you can restore it if required:
```
$ cp values.yaml /tmp/values.yaml
```
Edit the values.yaml file and enable volume snapshot:
1. Enable snapshot for ds-idrepo by setting the ds-idrepo.snapshot.enabled parameter to true.
  
  For example, change line 760 from enabled: false to enabled: true:
  ... 759 snapshot: 760 enabled: true 761 name: ds-idrepo-snapshot 762 pvcName: data-ds-idrepo-0 ...
2. Enable snapshot for ds-cts by setting the ds-cts.snapshot.enabled parameter to true.
  
  For example, change line 834 from enabled: false to enabled: true:
  ... 833 snapshot: 834 enabled: true 835 name: ds-cts-snapshot 836 pvcName: data-ds-cts-0 ...
Apply the changed value.yaml file:
```
$ helm upgrade -i
```

You can view the volume snapshots that are available for restore, using this command:

$ kubectl get volumesnapshots --namespace my-namespace

NAME                               READYTOUSE   SOURCEPVC          SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT
          CREATIONTIME   AGE
ds-idrepo-snapshot-20231117-1320   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-be3f4a44-cfb2-4f68-aa2b-60902
bb44192   3h29m          3h29m
ds-idrepo-snapshot-20231117-1330   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-7bcf6779-382d-40e3-9c9f-edf31
c54768e   3h19m          3h19m
ds-idrepo-snapshot-20231117-1340   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-c9c88332-ad05-4880-bda7-48616
ec13579   3h9m           3h9m
ds-idrepo-snapshot-20231117-1401   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-1f3f4ce9-0083-447f-9803-f6b45
e03ac27   167m           167m
ds-idrepo-snapshot-20231117-1412   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-4c39c095-0891-4da8-ae61-fac78
c7147ff   156m           156m

Customize the backup schedule

When enabled, volume snapshots are created once every day by default and purged after three days. You can customize the backup schedules as required in your environment.

In a Kustomize-based deployment
In a Helm-based deployment

To modify the default schedule and purge delay for the idrepo repository^[1]:

In a terminal window, change to the path/to/idrepo directory.
Copy the schedule.yaml file to a temporary location, so you can restore if needed.
Edit the schedule.yaml file and set the schedule and purge-delay parameters as needed.
Run the kubectl apply command.
Examples for scheduling snapshots
To schedule snapshots twice a day, at noon and midnight:

... spec: schedule: "0 0/12 * * *" ...

To schedule snapshots every 8 hours:

... spec: schedule: "0 */8 * * *" ...
Examples for purging schedule
To schedule purge after 4 days:

... env: - name: PURGE_DELAY value: "-4 day"

To schedule purge after a week:

... env: - name: PURGE_DELAY value: "-7 day"

To modify the default schedule and purge delay:

In a terminal window, change to the identity-platform subdirectory under the charts directory:
```
$ cd /path/to/forgeops/charts/identity-platform
```
Copy the values.yaml file, so you can restore it if required:
```
$ cp values.yaml /tmp/values.yaml
```
Edit the charts/identity-platform/values.yaml file and set the schedule and purge-delay parameters as needed.
Run the helm upgrade command.
Examples for scheduling snapshots for the idrepo repository^[2]
To schedule snapshots twice a day, at noon and midnight:

... ds-idrepo: ... snapshot: ... schedule: "0 0/12 * * *" ...

To schedule snapshots every 8 hours:

... ds-idrepo: ... snapshot: ... schedule: "0 */8 * * *" ...
Examples for purging schedule for the idrepo repository^[2]
To schedule purge after 4 days:

... ds-idrepo: ... snapshot: ... purgeDelay: "-4 day" ...

To schedule purge after a week:

... ds-idrepo: ... snapshot: ... purgeDelay: "-7 day" ...

Restore from volume snapshot

The snapshot-restore.sh script lets you restore DS instances in a ForgeOps deployment. By default, this script restores a DS instance from the latest available snapshot.

There are two options when using the snapshot-restore.sh script to restore a DS from a volume snapshot:

Full—Use the full option to fully restore a DS instance from a volume snapshot. When you specify this option, the DS is scaled down to 0 pods before restoring data. The data is restored to an existing PVC from a snapshot. This operation requires downtime.
Selective—Use the selective option to restore a portion of DS data from volume snapshot. The selective restore creates a new temporary DS instance with a new DS pod. You can selectively export from the temporary DS pod and import into your functional DS instance. After restoring data, you can clean up the temporary resources.

The snapshot-restore.sh command is available in the bin directory of the forgeops repository. To learn more about the snapshot-restore.sh command and its options, run snapshot-restore.sh --help.

Restore examples

Trial run without actually restoring DS data

In a terminal window, change to the /path/to/forgeops/bin directory.
Set your Kubernetes context to the correct cluster and namespace.

Run the snapshot-restore.sh command with the --dryrun option:

$ ./snapshot-restore.sh --dryrun --namespace my-namespace full idrepo

./snapshot-restore.sh --dryrun --namespace my-namespace full idrepo
/usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/sts-restore.json -n my-namespace
/usr/local/bin/kubectl delete pvc data-ds-idrepo-0 -n my-namespace
/usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/data-ds-idrepo-0.json -n my-namespace
/usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/sts.json -n my-namespace

Full restore of the idrepo instance from the latest available volume snapshot

In a terminal window, change to the /path/to/forgeops/bin directory.
Set your Kubernetes context to the correct cluster and namespace.

Get a list of available volume snapshots:

$ kubectl get volumesnapshots --namespace my-namespace

Restore the full DS instance:

$ ./snapshot-restore.sh --namespace my-namespace full idrepo

Verify that DS data has been restored.

Selective restore from a specific volume snapshot and storing data in a user-defined storage path

In a terminal window, change to the /path/to/forgeops/bin directory.
Set your Kubernetes context to the correct cluster and namespace.

Get a list of available volume snapshots:

$ kubectl get volumesnapshots --namespace my-namespace

Perform a selective restore trial run:

$ ./snapshot-restore.sh --dryrun --path /tmp/ds-restore --snapshot ds-idrepo-snapshot-20231121-2250 --namespace my-namespace selective idrepo

VolumeSnapshot ds-idrepo-snapshot-20231121-2250 is ready to use
/usr/local/bin/kubectl apply -f /tmp/ds-rest/sts-restore.json -n my-namespace
/usr/local/bin/kubectl apply -f /tmp/ds-rest/svc.json -n my-namespace

Perform a selective restore using a specific snapshot:

$ ./snapshot-restore.sh --path /tmp/ds-restore --snapshot ds-idrepo-snapshot-20231121-2250 --namespace my-namespace selective idrepo

statefulset.apps/ds-idrepo-restore created
service/ds-idrepo configured

Verify that a new ds-idrepo-restore-0 pod was created:

$ kubectl get pods
NAME                          READY   STATUS      RESTARTS   AGE
admin-ui-656db67f54-2brbf     1/1     Running     0          3h17m
am-7fffff59fd-mkks5           1/1     Running     0          107m
amster-hgkv9                  0/1     Completed   0          3h18m
ds-idrepo-0                   1/1     Running     0          39m
ds-idrepo-restore-0           1/1     Running     0          2m40s
end-user-ui-df49f79d4-n4q54   1/1     Running     0          3h17m
idm-fc88578bf-lqcdj           1/1     Running     0          3h18m
login-ui-5945d48fc6-ljxw2     1/1     Running     0          3h17m

The ds-idrepo-restore-0 pod is temporary and not to be used as a complete DS instance. You can export required data from the temporary pod, and import data into your functional DS instance.

Clean up resources from the selective restore:

$ ./snapshot-restore.sh clean idrepo

statefulset.apps "ds-idrepo-restore" deleted
persistentvolumeclaim "data-ds-idrepo-restore-0" deleted

1. Use similar steps to modify the schedule and purge delay for the cts repository

2. Change the ds-cts parameters to modify the schedule and purge delay for the cts repository