Backup and Restore

The backup topology of DS in the CDM:

Diagram about backup topology.

Some important DS backup and restore considerations in the CDM:

  • Six DS instances are deployed using Kubernetes stateful sets (three each for idrepo and cts backends).

  • DS data backups are stored in Google Cloud Service, Amazon S3, or Azure Blob Storage. Before deploying the CDM set up a cloud storage container and the necessary credentials to access the container.

  • Backups must be scheduled using the schedule-backups.sh script in the /path/to/forgeops/bin path on your local computer.

  • By default, an incremental backup of the CTS and identity repository directory instances is performed every hour. The first incremental backup created is a full backup.

  • The backup schedule can be customized separately for CTS and identity repository DS instances based on your recovery objectives.

  • By default, the backups are scheduled from the last (or -2) pod of each DS instance. You can customize and schedule backups from any pod of a DS instance. You can also schedule backups from multiple pods.

  • DS server instances use cryptographic keys to sign and verify the integrity of backup files, and to encrypt data. Server instances protect these keys in a deployment by encrypting them with the shared master key. For portability, servers store the encrypted keys in the backup files.

  • An idrepo instance can be restored using the backup of any other idrepo instance, as long as it holds a copy of the shared master key used to encrypt the keys.

  • Similarly, a cts instance can also be restored using the backup of another cts instance, as long as the shared master key is available to the cts being restored.

  • You can set up new DS instances with data from a previous backup when you deploy CDM.

DS Backup Schedules

After you deploy the CDM, schedule backups of the directory data.

Default Backup Schedule

The default backup schedule creates incremental backups of the idrepo instance at the beginning of every hour, and the cts instance at 10 minutes past every hour.

Run the schedule-backups.sh script to start backing up cts and idrepo instances using the default schedule:

$  /path/to/forgeops/bin/schedule-backups.sh my-namespace

In the CDM deployment, DS is deployed to the prod namespace. So you would specify prod as the namespace in the above command. If you have deployed DS in another namespace, you must specify the corresponding namespace.

Customized Backup Schedule

You can customize the backup schedule for cts and idrepo instances separately. You can also schedule backups from any DS pod.

To customize the schedule for the idrepo instance, and to schedule backups from the ds-idrepo-1 pod in addition to the ds-idrepo-2 pod:

Customize Backup Schedule

  1. Edit the schedule-backups.sh file in the /path/to/forgeops/bin folder and change the line BACKUP_SCHEDULE_IDREPO="0 * * * *" to suit your backup schedule for idrepo.

    For example, the following line schedules an incremental backup for the idrepo instance every 30 minutes:

    BACKUP_SCHEDULE_IDREPO="*/30 * * * *"

  2. Edit the kustomization.yaml file in the /path/to/forgeops/kustomize/base/kustomizeConfig folder on your local computer and change the line that begins with - DSBACKUP_HOSTS= to:

    - DSBACKUP_HOSTS="ds-cts-2,ds-idrepo-2,ds-idrepo-1"

  3. Run the schedule-backups.sh script.

    $ /path/to/forgeops/bin/schedule-backups.sh prod

Similarly, edit the schedule-backups.sh script and change the line BACKUP_SCHEDULE_CTS="10 * * * *" to suit your backup schedule for the cts instance. If you want to add or change the cts pod that performs backup, edit the kustomization.yaml file in the /path/to/forgeops/kustomize/base/kustomizeConfig folder. Then run the schedule-backups.sh script.

The schedule-backups.sh script stops any previously scheduled backup jobs before initiating the new schedule.

CDM Automatic Restore

When the CDM is deployed, new, empty DS instances are created. You can enable the automatic restore capability in CDM to:

  • Deploy new DS instances using the data from a previous backup.

  • Set up the cts and idrepo DS pods to recover from operational failures.

The automatic restore capability to deploy new instances with backup data is useful when a system disaster occurs, or when directory services are lost. It is also useful when you want to port test environment data to a production deployment. Note that when you restore a DS instance, all its replicas are restored.

You can enable automatic restore before you deploy CDM. To enable automatic restore, you must set up the DS data backup to cloud storage, copy the backup data to the cloud storage location, and then follow these steps:

Enable Automatic Restore

  1. Change to the /path/to/forgeops/kustomize/base/kustomizeConfig directory.

  2. Edit the kustomization.yaml file and set the AUTORESTORE_FROM_DSBACKUP parameter to "true".

  3. Deploy platform as described in CDM Deployment.

In a CDM deployment that has automatic restore enabled, you can recover a failed pod:

Recover a Failed DS Pod

  1. Delete the PVC attached to the failed DS pod using the kubectl delete pvc command. The PVC might not get deleted immediately if the attached pod is running.

  2. In another terminal window, stop the failed DS pod using the kubectl delete pod command. This will delete the PVC.

The automatic restore feature of CDM recreates the PVC, copies data from the backup, restores the DS pod with the latest backup, and replicates the current data from other DS pods.

You can also perform manual restore of DS. For more information about how to manually restore DS, see the Restore section in the ForgeRock Directory Services Maintenance Guide.

Consider following these best practices:

  • Use a backup that is newer than the last replication purge.

  • When you restore a DS replica using backups that are older than the purge delay, that replica will no longer be able to participate in replication. Reinitialize the replica to restore the replication topology.

  • If the available backups are older than the purge delay, then initialize the DS replica from an up-to-date master instance. For more information on how to initialize a replica, see Manual Initialization in the ForgeRock Directory Services Configuration Guide.