Data Storage

DS directory servers store data in backends. A backend is a private server repository that can be implemented in memory, as an LDIF file, or as an embedded database.

Embedded database backends have these characteristics:

Suitable for large numbers of entries

When creating a database backend, you choose the backend type. DS directory servers use JE backends for local data.

The JE backend type is implemented using B-tree data structures. It stores data as key-value pairs, which is different from the model used by relational databases.

JE backends are designed to hold hundreds of millions, or even billions of LDAP entries.

Fully managed by DS servers

Let the server manage its backends and their database files.

By default, backend database files are located under the opendj/db directory.

Warning

Do not compress, tamper with, or otherwise alter backend database files directly, unless specifically instructed to do so by a qualified ForgeRock technical support engineer. External changes to backend database files can render them unusable by the server.

If you use snapshot technology for backup, read "Back Up Using Snapshots" and "Restore From a Snapshot".

DS servers provide the dsbackup command for backing up and restoring database backends. For details, see Backup and Restore.

Self-cleaning

A JE backend stores data on disk using append-only log files with names like number.jdb. The JE backend writes updates to the highest-numbered log file. The log files grow until they reach a specified size (default: 1 GB). When the current log file reaches the specified size, the JE backend creates a new log file.

To avoid an endless increase in database size on disk, JE backends clean their log files in the background. A cleaner thread copies active records to new log files. Log files that no longer contain active records are deleted.

Due to the cleanup processes, JE backends can actively write to disk, even when there are no pending client or replication operations.

Configurable I/O behavior

By default, JE backends let the operating system potentially cache data for a period of time before flushing the data to disk. This setting trades full durability with higher disk I/O for good performance with lower disk I/O.

With this setting, it is possible to lose the most recent updates that were not yet written to disk, in the event of an underlying OS or hardware failure.

You can modify this behavior by changing the advanced configuration settings for the JE backend. If necessary, you can change the advanced setting, db-durability, using the dsconfig set-backend-prop command.

Automatic recovery

When a JE backend is opened, it recovers by recreating its B-tree structure from its log files.

This is a normal process. It lets the backend recover after an orderly shutdown or after a crash.

Automatic verification

JE backends run checksum verification periodically on the database log files.

If the verification process detects backend database corruption, then the server logs an error message and takes the backend offline. If this happens, restore the corrupted backend from backup so that it can be used again.

By default, the verification runs every night at midnight local time. If necessary, you can change this behavior by adjusting the advanced settings, db-run-log-verifier and db-log-verifier-schedule, using the dsconfig set-backend-prop command.

Encryption support

JE backends support encryption for data confidentiality.

For details, see Data Encryption.

Depending on the setup profiles used at installation time, DS servers have backends with the following default settings:

Backend	Type	Optional?^[a]	Replicated?	Part of Backup?	Details
adminRoot	LDIF	No	If enabled	If enabled	Symmetric keys for (deprecated) reversible password storage schemes. Base DN: `cn=admin data` Base directory: `db/adminRoot`
amCts	JE	Yes	Yes	Yes	AM core token store (CTS) data. More information: Install DS for AM CTS. Base DN: `ou=tokens` Base directory: `db/amCts`
amIdentityStore	JE	Yes	Yes	Yes	AM identities. More information: Install DS for AM Identities. Base DN: `ou=identities` Base directory: `db/amIdentityStore`
cfgStore	JE	Yes	Yes	Yes	AM configuration, policy, and other data. More information: Install DS for AM Configuration. Base DN: `ou=am-config` Base directory: `db/cfgStore`
changelogDb	Changelog	Yes^[b]	N/A	No	Change data for replication and change notifications. More information: Changelog for Notifications. Base DN: `cn=changelog` Base directory: `changelogDb`
config	Config	No	N/A	No	File-based representation of this server's configuration. Do not edit `config/config.ldif` directly. Use the dsconfig command instead. Base DN: `cn=config` Base directory: `config`
dsEvaluation	JE	Yes	Yes	Yes	Example.com sample data. More information: Install DS for Evaluation. Base DN: `dc=example,dc=com` Base directory: `db/dsEvaluation`
idmRepo	JE	Yes	Yes	Yes	IDM repository data. More information: Install DS as an IDM Repository. Base DN: `dc=openidm,dc=forgerock,dc=com` Base directory: `db/idmRepo`
monitor	Monitor	No	N/A	N/A	Single entry; volatile monitoring metrics maintained in memory since server startup. More information: LDAP-Based Monitoring. Base DN: `cn=monitor` Base directory: N/A
monitorUser	LDIF	Yes	Yes	Yes	Single entry; default monitor user account. Base DN: `uid=Monitor` Base directory: `db/monitorUser`
rootUser	LDIF	No^[c]	No	Yes	Single entry; default directory superuser account. Base DN: `uid=admin` Base directory: `db/rootUser`
root DSE	Root DSE	No	N/A	N/A	Single entry describing server capabilities. Use ldapsearch --baseDn "" --searchScope base "(&)" + to read all the (operational) attributes of this entry. Base DN: `""` (empty string) Base directory: N/A
schema	Schema	No	Yes	Yes	Single entry listing LDAP schema definitions. More information: LDAP Schema. Base DN: `cn=schema` Base directory: `db/schema`
tasks	Task	No	N/A	Yes	Scheduled tasks for this server. Use the manage-tasks command. Base DN: `cn=tasks` Base directory: `db/tasks`
userData	JE	Yes	Yes	Yes	User entries imported from the LDIF you provide. More information: Install DS for User Data. Base directory: `db/userData`
^[a] Optional backends depend on the setup choices. ^[b] The changelog backend is mandatory for servers with a replication server role. ^[c] You must create a superuser account at setup time. You may choose to replace it later. For details, see "Use a Non-Default Superuser Account".

Create a Backend

Configure the new backend.

The following example creates a database backend for Example.org data. The backend relies on a JE database for data storage and indexing:

$ dsconfig \
 create-backend \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name exampleOrgBackend \
 --type je \
 --set enabled:true \
 --set base-dn:dc=example,dc=org \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --no-prompt

When you create a new backend using the dsconfig command, DS directory servers create the following indexes automatically:

Index	Approx.	Equality	Ordering	Presence	Substring	Entry Limit
`aci`	-	-	-	Yes	-	4000
`dn2id`	Non-configurable internal index
`ds-sync-conflict`	-	Yes	-	-	-	4000
`ds-sync-hist`	-	-	Yes	-	-	4000
`entryUUID`	-	Yes	-	-	-	4000
`id2children`	Non-configurable internal index
`id2subtree`	Non-configurable internal index
`objectClass`	-	Yes	-	-	-	4000

Let the server replicate the new backend:

$ dsconfig \
 create-replication-domain \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --provider-name "Multimaster Synchronization" \
 --domain-name dc=example,dc=org \
 --type generic \
 --set enabled:true \
 --set base-dn:dc=example,dc=org \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --no-prompt

Import and Export

Import LDIF

Important

Importing from LDIF overwrites all data in the target backend with entries from the LDIF data.
If the LDIF was exported from another server, it may contain pre-encoded passwords. By default, password policies do not allow you to use pre-encoded passwords. You can change this behavior by changing the default password policy configuration property, allow-pre-encoded-passwords.
LDIF from another server can include passwords encrypted with a reversible storage scheme, such as AES or Blowfish. To decrypt the passwords, the server must use the same deployment key as the server that encrypted the passwords.

The following example imports dc=example,dc=com data into the dsEvaluation backend, overwriting existing data:
- To speed up the process when you have millions of directory entries to import, first shut down the server, and then run the import-ldif command:
```
$ stop-ds
$ import-ldif \
 --offline \
 --backendId dsEvaluation \
 --includeBranch dc=example,dc=com \
 --ldifFile generated.ldif
```
- To run the import while the server is online, schedule a task:
```
$ start-ds
$ import-ldif \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backendId dsEvaluation \
 --includeBranch dc=example,dc=com \
 --ldifFile generated.ldif \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin
```
  The import task is requested on a secure connection to the administration port, such as 4444. You can schedule the import task to start at a particular time using the --start option.
If the server is replicated with other servers, initialize replication again after the successful import.
For details see "Manual Initialization".
Initializing replication overwrites data in the remote servers like import overwrites existing data with LDIF data.

Export LDIF

The following examples export dc=example,dc=com data from the dsEvaluation backend:

To expedite export, shut down the server and then use the export-ldif command:

$ stop-ds
$ export-ldif \
 --offline \
 --backendId dsEvaluation \
 --includeBranch dc=example,dc=com \
 --ldifFile backup.ldif

To export the data while online, schedule a task:
```
$ export-ldif \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backendId dsEvaluation \
 --includeBranch dc=example,dc=com \
 --ldifFile backup.ldif \
 --start 0 \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin
```
The --start 0 option tells the directory server to start the export task immediately.
You can specify a time for the task to start using the format yyyymmddHHMMSS. For example, 20250101043000 specifies a start time of 4:30 AM on January 1, 2025.
If the server is not running at the time specified, it attempts to perform the task after it restarts.

Split Data

Splitting data impacts replication, and can require that you interrupt the service.

One one set of replicas, create the exampleData backend to hold all dc=example,dc=com data except ou=People,dc=example,dc=com:

# Create a backend for the data not under the subordinate base DN:
$ dsconfig \
 create-backend \
 --backend-name exampleData \
 --type je \
 --set enabled:true \
 --set base-dn:dc=example,dc=com \
 --offline \
 --no-prompt

# Let the server replicate the new backend:
$ dsconfig \
 create-replication-domain \
 --provider-name "Multimaster Synchronization" \
 --domain-name dc=example,dc=com \
 --type generic \
 --set enabled:true \
 --set base-dn:dc=example,dc=com \
 --offline \
 --no-prompt

# Import data not under the subordinate base DN:
$ import-ldif \
 --backendId exampleData \
 --excludeBranch ou=People,dc=example,dc=com \
 --ldifFile generated.ldif \
 --offline

On another set of replicas, create a peopleData backend to hold ou=People,dc=example,dc=com data:

# Create a backend for the data under the subordinate base DN:
$ dsconfig \
 create-backend \
 --backend-name peopleData \
 --type je \
 --set enabled:true \
 --set base-dn:ou=People,dc=example,dc=com \
 --offline \
 --no-prompt

# Let the server replicate the new backend:
$ dsconfig \
 create-replication-domain \
 --provider-name "Multimaster Synchronization" \
 --domain-name ou=People,dc=example,dc=com \
 --type generic \
 --set enabled:true \
 --set base-dn:ou=People,dc=example,dc=com \
 --offline \
 --no-prompt

# Import data under the subordinate base DN:
$ import-ldif \
 --backendId peopleData \
 --includeBranch ou=People,dc=example,dc=com \
 --ldifFile generated.ldif \
 --offline

After completing the data import process, start the replicas.

To split an existing backend, follow these high-level steps:

Stop the affected replica.
Create the new backend.
Export the data for the backend.
Import the data under the subordinate base DN into the new backend, using the --includeBranch option.
Delete all data under the subordinate base DN from the old backend.
Start the affected replica.

Disk Space Thresholds

Directory data growth depends on applications that use the directory. When directory applications add more data than they delete, the database backend grows until it fills the available disk space. The system can end up in an unrecoverable state if no disk space is available.

Database backends therefore have advanced properties, disk-low-threshold and disk-full-threshold. When available disk space falls below disk-low-threshold, the directory server only allows updates from users and applications that have the bypass-lockdown privilege. When available space falls below disk-full-threshold, the directory server stops allowing updates, instead returning an UNWILLING_TO_PERFORM error to each update request.

If growth across the directory service tends to happen quickly, set the thresholds higher than the defaults to allow more time to react when growth threatens to fill the disk. The following example sets disk-low-threshold to 10 GB disk-full-threshold to 5 GB for the dsEvaluation backend:

$ dsconfig \
 set-backend-prop \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name dsEvaluation \
 --set "disk-low-threshold:10 GB" \
 --set "disk-full-threshold:5 GB" \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --no-prompt

The properties disk-low-threshold and disk-full-threshold are listed as advanced properties. To examine their values with the dsconfig command, use the --advanced option:

$ dsconfig \
 get-backend-prop \
 --advanced \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name dsEvaluation \
 --property disk-low-threshold \
 --property disk-full-threshold \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --no-prompt
Property            : Value(s)
--------------------:---------
disk-full-threshold : 5 gb
disk-low-threshold  : 10 gb

Entry Expiration

If the directory service creates many entries that expire and should be deleted, you could finding the entries with a time-based search and then delete them individually. That approach uses replication to delete the entries in all replicas. It has the disadvantage of generating potentially large amounts of replication traffic.

Entry expiration lets you configure the backend database to delete the entries as they expire. This approach deletes expired entries at the backend database level, without generating replication traffic. AM uses this approach when relying on DS to delete expired token entries, as demonstrated in Install DS for AM CTS.

Backend indexes for generalized time (timestamp) attributes have these properties to configure automated, optimized entry expiration and removal:

Configure this capability by performing the following steps:

Prepare an ordering index for a generalized time (timestamp) attribute on entries that expire.
For details, see "Configure Indexes" and "Active Accounts".
Using the dsconfig set-backend-index-prop command, set ttl-enabled on the index to true, and set ttl-age on the index to the desired entry lifetime duration.

Once you configure and build the index, the backend can delete expired entries. At intervals of 10 seconds, the backend automatically deletes entries whose timestamps on the attribute are older than the specified lifetime. Entries that expire in the interval between deletions are removed on the next round. Client applications should therefore check that entries have not expired, as it is possible for expired entries to remain available until the next round of deletions.

When using this capability, keep the following points in mind:

Entry expiration is per index. The time to live depends on the value of the indexed attribute and the ttl-age setting, and all matching entries are subject to TTL.
If multiple indexes' ttl-enabled and ttl-age properties are configured, as soon as one of the entry's matching attributes exceeds the TTL, the entry is eligible for deletion.
The backend deletes the entries directly. Deletion is not visible at the protocol level, and so is not recorded in logs nor returned in persistent searches.
Furthermore, this means that deletion is not replicated. To ensure expired entries are deleted on all replicas, use the same indexes with the same settings on all replicas.
When a backend deletes an expired entry, the effect is a subtree delete. In other words, if a parent entry expires, the parent entry and all the parent's child entries are deleted.
If you do not want parent entries to expire, index a generalized time attribute that is only present on its child entries.
The backend deletes expired entries atomically.
If you update the TTL attribute to prevent deletion and the update succeeds, then TTL has effectively been reset.
Expiration fails when the index-entry-limit is exceeded. (For background information, see "Index Entry Limits".)
This only happens if the timestamp for the indexed attribute matches to the nearest millisecond on more than 4000 entries (for default settings). This corresponds to four million timestamp updates per second, which would be very difficult to reproduce in a real directory service.
It is possible, however, to construct and import an LDIF file where more than 4000 entries have the same timestamp. Make sure not to reuse the same timestamp for thousands of entries when artificially creating entries that you intend to expire.

Add a Base DN

The following example adds the base DN o=example to the dsEvaluation backend:

$ dsconfig \
 set-backend-prop \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name dsEvaluation \
 --add base-dn:o=example \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --no-prompt
$ dsconfig \
 get-backend-prop \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name dsEvaluation \
 --property base-dn \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePasswordFile /path/to/opendj/config/keystore.pin \
 --no-prompt
Property : Value(s)
---------:-------------------------------
base-dn  : "dc=example,dc=com", o=example

When you add a base DN to a backend, the base DN must not be subordinate to any other base DNs in the backend. As in the commands shown here, you can add the base DN o=example to a backend that already has a base DN dc=example,dc=com. You cannot, however, add o=example,dc=example,dc=com as a base DN, because that is a child of dc=example,dc=com.

Delete a Backend

When you delete a database backend with the dsconfig delete-backend command, the directory server does not actually remove the database files for these reasons:

A mistake could potentially cause lots of data to be lost.
Deleting a large database backend could cause severe service degradation due to a sudden increase in I/O load.

After you run the dsconfig delete-backend command, manually remove the database backend files.

If run the dsconfig delete-backend command by mistake, but have not yet deleted the actual files, recover the data by creating the backend again, and reconfiguring and rebuilding the indexes.