How do I design and implement my backup and restore strategies for DS (All versions)?

Overview

Each DS integration is different and the backup strategies must be evaluated based on your business requirements.

There are also changes to the mechanism for backups and restore, as well as data encryption, depending on your DS version:

• DS 7 and later

DS 7 introduced a simplified implementation for backup and restore operations as explained in New in 7.0.0 (Backup and Restore), but many of the strategies discussed here are still applicable.

Additionally, DS 7 and later servers are secure by default, which means that when you are restoring backups, the server must have access to the master key pair used to encrypt and sign the files when they were created in order to decrypt and verify signatures on the backup files.

• DS 6.x

If you have set up your servers in Production Mode, you should be aware that the default backend database (userRoot) uses data confidentiality to encrypt potentially sensitive data on disk, which means any restores need to be able to decrypt the confidential data. See Encrypting Directory Data for further information.

Backup Strategies

Full Backups

Full backups are only relevant to DS 6.x because all backups in DS 7 and later are incremental.

Full backups are normally done on a scheduled basis, but they are considered a Point In Time (PIT) backup that is not 100% up to date.

Considerations:

• Take the DS instance out of the load balancer until the backup is complete.
• Include the Other Required Files by manually copying these files to a safe location. Although all these files are important, admin-backend.ldif is critical if you set up the server in Production Mode because it is needed to decrypt the backend data. If you do not have all the keys in the admin backend, you will not be able to access the data and DS will not start. The keystore and keystore.pin files are essential to back up for a similar reason regardless of whether you are using Production Mode or not.
• Ensure you have enough disk space to store sufficient backup snapshots to adhere to your business requirements.
• Consider adding Incremental Backups to augment the full backup. This will allow for a more complete restore point, but can take more time as each incremental backup is restored.

Frequency:

• It is recommended to create full backups at least once a week with daily incremental backups.
• It is recommended to create full backups at least once per purge delay interval with incremental backups more frequently.

Incremental Backups

Incremental backups can be used exclusively to back up data over time, but each and every incremental backup from the point of the last full backup must be applied on top of each other to fully restore the database to a usable state.

Considerations (DS 6.x only):

• Consider using a daily full backup followed by incremental backups every six hours or less.

Frequency:

• Daily incremental backups are recommended.
• The frequency can be increased if the business requirements stipulate more up-to-date backups are needed.

Expectations:

When entry counts are small, incremental backups may appear to look like full backups in the logs.

In the following example, the backup contained only one database file and it was thought the incremental backup looked more like a full backup. In truth, the entry counts were small and only one database jdb file was being used:

The number of jdb files present can vary depending on the DS version you are using, the number of entries in the database and the size of the entries. The default database size (db-log-file-max) is 1 GB.

See Database Cache Settings for further information.

System Backups

Considerations:

• Full system-level backups can be used in place of full backups (DS 6.x).
• You should stop the DS instance before doing a full system backup. This ensures the database and relevant files have been properly closed at the OS level.

Frequency:

• It is recommended you create full system-level backups at least once a week.

Backup Commands

DS comes with the following backup and export commands that can be used to back up data from the DS instance:

Backup

• dsbackup (DS 7 and later): The dsbackup create command can be used:
  • For ad hoc or scheduled backups.
  • To back up individual or all backends.
• backup (DS 6.x): The DS backup command can be used:
  • For ad hoc or scheduled backups.
  • To back up individual or all backends.
  • To back up data in full or incrementally.

See Backup and Restore (DS 7 and later) or Backing Up Directory Data (DS 6.x) for further information.

Export

The export-ldif command allows administrators to extract a text-based copy of a backend in the form of an LDIF (Lightweight Data Interchange Format) file.

See Import and Export (DS 7 and later) or Importing and Exporting Data (DS 6.x) for further information.

Files Backed Up

dsbackup (DS 7 and later)

Example of files captured with the dsbackup create command:-rw-r--r-- 1 opendj opendj 57595457 May 24 08:47 00000000.jdb_opendj1_dsEvaluation_20220523184035_140563664.dsbk -rw-r--r-- 1 opendj opendj 2749 May 24 08:47 00000000.jdb_opendj1_dsEvaluation_20220523184035_140563664.info -rw-r--r-- 1 opendj opendj 1558 May 24 08:47 dsEvaluation_20220524144732555.idxWhere:

• Each backup has a .dsbk file, which contains the encrypted backup data and a corresponding backup info file (.info).
• Each backend backup has an .idx file, which you can view with a text editor and it will show you the corresponding backup files, for example:$ cat dsEvaluation_20220524144732555.idx . . . backendId: dsEvaluation fileIds: 00000000.jdb_opendj1_dsEvaluation_20220523184035_140563664

backup (DS 6.x)

Files captured with the backup command can include jdb database files, schema files and task ldif files.

• Database files: userRoot backend: -rw-r--r-- 1 opendj opendj 99999973 Jan 19 14:54 00000000.jdb -rw-r--r-- 1 opendj opendj 99999797 Jan 19 14:55 00000001.jdb -rw-r--r-- 1 opendj opendj 99999864 Jan 19 14:55 00000002.jdb -rw-r--r-- 1 opendj opendj 99997251 Jan 19 14:55 00000003.jdb
• Schema files: schema backend: -rw-r--r-- 1 opendj opendj 43693 Jan 19 11:04 00-core.ldif -rw-r--r-- 1 opendj opendj 6596 Jan 19 11:04 01-pwpolicy.ldif -rw-r--r-- 1 opendj opendj 194661 Jan 19 11:04 02-config.ldif -rw-r--r-- 1 opendj opendj 5058 Jan 19 11:04 03-changelog.ldif -rw-r--r-- 1 opendj opendj 1227 Jan 19 11:04 03-pwpolicyextension.ldif -rw-r--r-- 1 opendj opendj 3561 Jan 19 11:04 03-rfc2713.ldif -rw-r--r-- 1 opendj opendj 2133 Jan 19 11:04 03-rfc2714.ldif -rw-r--r-- 1 opendj opendj 3235 Jan 19 11:04 03-rfc2739.ldif -rw-r--r-- 1 opendj opendj 3272 Jan 19 11:04 03-rfc2926.ldif -rw-r--r-- 1 opendj opendj 1710 Jan 19 11:04 03-rfc3112.ldif -rw-r--r-- 1 opendj opendj 12652 Jan 19 11:04 03-rfc3712.ldif -rw-r--r-- 1 opendj opendj 15858 Jan 19 11:04 03-uddiv3.ldif -rw-r--r-- 1 opendj opendj 12207 Jan 19 11:04 04-rfc2307bis.ldif -rw-r--r-- 1 opendj opendj 6339 Jan 19 11:04 05-rfc4876.ldif -rw-r--r-- 1 opendj opendj 12223 Jan 19 11:04 05-samba.ldif -rw-r--r-- 1 opendj opendj 14141 Jan 19 11:04 05-solaris.ldif -rw-r--r-- 1 opendj opendj 1125 Jan 19 11:04 06-compat.ldif -rw-r--r-- 1 opendj opendj 122837 Jan 19 12:09 99-user.ldif
• Scheduled Tasks: tasks backend: -rw------- 1 opendj opendj 572 Jan 19 11:05 tasks.ldif

export-ldif

The export-ldif command is used to export entry data from the DS database backend into a text-based file called LDIF. The export-ldif command extracts the binary data from the *.jdb database files and saves it as an LDIF file.

Format of Files Backed Up

Each backup command saves the files backed up in various ways.

dsbackup (DS 7 and later)

All backup files are stored as shown above in the selected backup location. They are not contained within a zip file.

backup (DS 6.x)

All files with each backend backup are contained within a compressed .zip file. Other than a schema backend backup, all files within the backup itself use the original file names.

• Database files: userRoot backend: backup-userRoot-20220204231456Z: Zip archive data, at least v2.0 to extract userRoot/$ unzip -l backup-userRoot-20220204231456Z Archive: backup-userRoot-20220204231456Z OpenDJ backup 20220204231456Z of backend userRoot Length Date Time Name -------- ---- ---- ---- 6849913 02-04-22 16:14 00000000.jdb -------- ------- 6849913 1 file
• Schema files: schema backend: schema-backup-20220204231456Z: Zip archive data, at least v2.0 to extract schema/$ unzip -l schema-backup-20220204231456Z Archive: schema-backup-20220204231456Z OpenDJ schema backup 20220204231456Z Length Date Time Name -------- ---- ---- ---- 0 02-04-22 16:14 schema.comment 43693 02-04-22 16:14 00-core.ldif.instance 6596 02-04-22 16:14 01-pwpolicy.ldif.instance 194661 02-04-22 16:14 02-config.ldif.instance 5058 02-04-22 16:14 03-changelog.ldif.instance 1227 02-04-22 16:14 03-pwpolicyextension.ldif.instance 3561 02-04-22 16:14 03-rfc2713.ldif.instance 2133 02-04-22 16:14 03-rfc2714.ldif.instance 3235 02-04-22 16:14 03-rfc2739.ldif.instance 3272 02-04-22 16:14 03-rfc2926.ldif.instance 1710 02-04-22 16:14 03-rfc3112.ldif.instance 12652 02-04-22 16:14 03-rfc3712.ldif.instance 15858 02-04-22 16:14 03-uddiv3.ldif.instance 12207 02-04-22 16:14 04-rfc2307bis.ldif.instance 6339 02-04-22 16:14 05-rfc4876.ldif.instance 12223 02-04-22 16:14 05-samba.ldif.instance 14141 02-04-22 16:14 05-solaris.ldif.instance 1125 02-04-22 16:14 06-compat.ldif.instance -------- ------- 339691 18 files
• Scheduled Tasks: tasks backend: tasks-backup-20220204231456Z: Zip archive data, at least v2.0 to extract tasks/$ unzip -l tasks-backup-20220204231456Z Archive: tasks-backup-20220204231456Z OpenDJ tasks backup 20220204231456Z Length Date Time Name -------- ---- ---- ---- 1277 02-04-22 16:14 tasks.ldif -------- ------- 1277 1 file

ldif-export

As mentioned, export-ldif saves a copy of the database in a text-based file in LDIF format:

Example of an LDIF export (truncated):

Other Required Files

While the dsbackup/backup and export-ldif commands are good for backing up database backends, they miss crucial data points that are required to fully restore instances to their working state. It is recommended that you include each of the following (as needed) that are most relevant to your integration.

Critical files

• config.ldif - contains the main configuration for the DS instance, including replication data. This file is located in /path/to/ds/config.
• admin-backend.ldif - contains the certificate for each replicated instance and the admin entry used for replication. The admin backend "cn=admin data" is created at startup using this file. This file is located in /db/adminRoot.

The admin-backend.ldif file is especially critical if you set up the server in Production Mode because it is needed to decrypt the backend data. If you do not have all the keys in the admin backend, you will not be able to access the data and the DS will be rendered unusable.

• keystore/truststores - contains client (truststore and keystore), replication (ads-truststore - DS 6.x only) and administration (admin-truststore and admin-keystore) key pairs. These files are located in /path/to/ds/config; ads-truststore is located in /path/to/ds/db/ads-truststore for new installs.
• java.properties - contains the java properties used by the various command lines when executed. This file is located in /path/to/ds/config.

Non-critical files

• http-config.json - contains the REST2LDAP configuration. This file is located in /path/to/ds/config.
• wordlist.txt - contains all the words that all password policies use to match against when adding or changing passwords. This file is located in /path/to/ds/config.
• Init scripts - all init.d scripts used to start and stop DS instances when the system boots or shuts down.

Restoration Strategies

Production Mode (DS 6.x)

If you set up the server in Production Mode, the default backend database (userRoot) uses data confidentiality (is encrypted), which means it needs to be successfully decrypted on restore for it to remain operational. To achieve this, you need to restore the admin-backend.ldif file and admin backend from the same topology. If you do not have all the keys and a properly configured admin backend from the same topology, the data from the backup cannot be decrypted and is unusable.

The following considerations regarding replicated servers and different types of restores are also still applicable.

Replicated Servers

Special considerations must be taken when restoring replicated servers. See the following links for further information:

• DS 7 and later: See Restore for important information regarding restoring a replicated backend.
• DS 6.x: See How do I restore old backup data to a DS 6.x replication topology? and To Restore a Replica for further information on the process of restoring replicated servers.

Considerations: (*)

• Use a backup that is newer than the replication-purge-delay (3 days by default).
• If the available backups are older than the purge delay, initialization from an up-to-date Master is a better option if the data is small (several million entries); otherwise you should consider backup and restore from the up-to-date Master instead.
• Restores using backups that are older than the purge delay will succeed but there will be a gap in the replicated changes as discussed in the documentation links above and the server will need to be initialized.

Full Restores (DS 6.x)

Considerations:

• Other than the Purge Delay restriction above (*), no special considerations are needed.
• Offline (server stopped) restores will complete faster than online (server started) restores.

Incremental Restores

In DS 7 and later, restoring is much simpler since the new implementation only restores one version of each file in the backup directory, whereas the previous implementation restored files from the full backup archive, and then restored files from each incremental backup archive. This means Incremental Restores must be completed in the following way in DS 6.x:

Example: If a full backup was taken on Sunday and incremental backups were taken daily, Monday through Saturday, the administrator would have 7 backups to restore the instance up to the latest restore point. This should be done as follows:

1. Restore the latest full backup.
2. Restore each incremental backup starting with Monday and ending with Saturday's backup.

Considerations:

• Other than the Purge Delay restriction above (*), no special considerations are needed.
• Offline (server stopped) restores will complete faster than online (server started) restores.

System Restores

Considerations:

• Other than the Purge Delay restriction above (*), no special considerations are needed.

Bare Metal or Ground Zero Restores

In DS 7 and later, see Disaster Recovery for the steps involved.

DS 6.x: When restoring to a newly installed instance, the instance must have been configured with the ./setup command before the restore takes place. Failing to first configure the new unzipped instance and issuing the restore command will result in the following error and the restore will fail:

The following process must be adhered to in DS 6.x if you need to build a new server or restore a server from scratch:

1. Install and configure the instance to a basic level, that is, using the setup command.
2. Execute the restore command with the appropriate --backupID.

Considerations:

• Other than the Purge Delay restriction above (*), no special considerations are needed.

Point In Time Restores

If you need to restore all servers to a certain point in time you can do so as described in Restore to a Known State (DS 7 and later) or How do I roll back an entire network of DS 6.x replicas to a previous backup?

Considerations:

• Other than the Purge Delay restriction above (*), no special considerations are needed.

See Also

How do I configure DS 6.x to ensure accidentally deleted or changed data can be restored when replication is enabled?

Generation IDs do not match error after restoring a DS (All versions) replica

FAQ: Backup and restore in DS 6.x

How do I understand what the different DS Backends are used for in DS 6.x?

Backup and Restore (DS 7 and later)

Data Storage (DS 7)

Set Up Servers in Production Mode (DS 6.x)

Backing Up and Restoring Data (DS 6.x)

Related Training

N/A

Related Issue Tracker IDs

N/A