DS 7.1.8

Data Storage

DS directory servers store data in backends . A backend is a private server repository that can be implemented in memory, as an LDIF file, or as an embedded database.

Embedded database backends have these characteristics:

Suitable for large numbers of entries

When creating a database backend, you choose the backend type. DS directory servers use JE backends for local data.

The JE backend type is implemented using B-tree data structures. It stores data as key-value pairs, which is different from the model used by relational databases.

JE backends are designed to hold hundreds of millions, or even billions of LDAP entries.

Fully managed by DS servers

Let the server manage its backends and their database files.

By default, backend database files are located under the opendj/db directory.

Do not compress, tamper with, or otherwise alter backend database files directly, unless specifically instructed to do so by a qualified ForgeRock technical support engineer. External changes to backend database files can render them unusable by the server.

If you use snapshot technology for backup, read Back Up Using Snapshots and Restore From a Snapshot.

DS servers provide the dsbackup command for backing up and restoring database backends. For details, see Backup and Restore.

Self-cleaning

A JE backend stores data on disk using append-only log files with names like number.jdb. The JE backend writes updates to the highest-numbered log file. The log files grow until they reach a specified size (default: 1 GB). When the current log file reaches the specified size, the JE backend creates a new log file.

To avoid an endless increase in database size on disk, JE backends clean their log files in the background. A cleaner thread copies active records to new log files. Log files that no longer contain active records are deleted.

Due to the cleanup processes, JE backends can actively write to disk, even when there are no pending client or replication operations.

Configurable I/O behavior

By default, JE backends let the operating system potentially cache data for a period of time before flushing the data to disk. This setting trades full durability with higher disk I/O for good performance with lower disk I/O.

With this setting, it is possible to lose the most recent updates that were not yet written to disk, in the event of an underlying OS or hardware failure.

You can modify this behavior by changing the advanced configuration settings for the JE backend. If necessary, you can change the advanced setting, db-durability, using the dsconfig set-backend-prop command.

Automatic recovery

When a JE backend is opened, it recovers by recreating its B-tree structure from its log files.

This is a normal process. It lets the backend recover after an orderly shutdown or after a crash.

Automatic verification

JE backends run checksum verification periodically on the database log files.

If the verification process detects backend database corruption, then the server logs an error message and takes the backend offline. If this happens, restore the corrupted backend from backup so that it can be used again.

By default, the verification runs every night at midnight local time. If necessary, you can change this behavior by adjusting the advanced settings, db-run-log-verifier and db-log-verifier-schedule, using the dsconfig set-backend-prop command.

Encryption support

JE backends support encryption for data confidentiality.

For details, see Data Encryption.

Depending on the setup profiles used at installation time, DS servers have backends with the following default settings:

Backend Type Optional?(1) Replicated? Part of Backup? Details

adminRoot

LDIF

No

If enabled

If enabled

Symmetric keys for (deprecated) reversible password storage schemes.

Base DN: cn=admin data

Base directory: db/adminRoot

amCts

JE

Yes

Yes

Yes

AM core token store (CTS) data.

More information: Install DS for AM CTS.

Base DN: ou=tokens

Base directory: db/amCts

amIdentityStore

JE

Yes

Yes

Yes

AM identities.

More information: Install DS for AM Identities.

Base DN: ou=identities

Base directory: db/amIdentityStore

cfgStore

JE

Yes

Yes

Yes

AM configuration, policy, and other data.

More information: Install DS for AM Configuration.

Base DN: ou=am-config

Base directory: db/cfgStore

changelogDb

Changelog

Yes(2)

N/A

No

Change data for replication and change notifications.

More information: Changelog for Notifications.

Base DN: cn=changelog

Base directory: changelogDb

config

Config

No

N/A

No

File-based representation of this server’s configuration.

Do not edit config/config.ldif directly. Use the dsconfig command instead.

Base DN: cn=config

Base directory: config

dsEvaluation

JE

Yes

Yes

Yes

Example.com sample data.

More information: Install DS for Evaluation.

Base DN: dc=example,dc=com

Base directory: db/dsEvaluation

idmRepo

JE

Yes

Yes

Yes

IDM repository data.

More information: Install DS as an IDM Repository.

Base DN: dc=openidm,dc=forgerock,dc=com

Base directory: db/idmRepo

monitor

Monitor

No

N/A

N/A

Single entry; volatile monitoring metrics maintained in memory since server startup.

More information: LDAP-Based Monitoring.

Base DN: cn=monitor

Base directory: N/A

monitorUser

LDIF

Yes

Yes

Yes

Single entry; default monitor user account.

Base DN: uid=Monitor

Base directory: db/monitorUser

rootUser

LDIF

No(3)

No

Yes

Single entry; default directory superuser account.

Base DN: uid=admin

Base directory: db/rootUser

root DSE

Root DSE

No

N/A

N/A

Single entry describing server capabilities.

Use ldapsearch --baseDn "" --searchScope base "(&)" + to read all the (operational) attributes of this entry.

Base DN: "" (empty string)

Base directory: N/A

schema

Schema

No

Yes

Yes

Single entry listing LDAP schema definitions.

More information: LDAP Schema.

Base DN: cn=schema

Base directory: db/schema

tasks

Task

No

N/A

Yes

Scheduled tasks for this server.

Use the manage-tasks command.

Base DN: cn=tasks

Base directory: db/tasks

userData

JE

Yes

Yes

Yes

User entries imported from the LDIF you provide.

More information: Install DS for User Data.

Base directory: db/userData

(1) Optional backends depend on the setup choices.

(2) The changelog backend is mandatory for servers with a replication server role.

(3) You must create a superuser account at setup time. You may choose to replace it later. For details, see Use a Non-Default Superuser Account.

Create a Backend

When you create a new backend on a replicated directory server, let the server replicate the new data:

  1. Configure the new backend.

    The following example creates a database backend for Example.org data. The backend relies on a JE database for data storage and indexing:

    $ dsconfig \
 create-backend \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name exampleOrgBackend \
 --type je \
 --set enabled:true \
 --set base-dn:dc=example,dc=org \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

    When you create a new backend using the dsconfig command, DS directory servers create the following indexes automatically:

    Index Approx. Equality Ordering Presence Substring Entry Limit

    aci

    -

    -

    -

    Yes

    -

    4000

    dn2id

    Non-configurable internal index

    ds-sync-conflict

    -

    Yes

    -

    -

    -

    4000

    ds-sync-hist

    -

    -

    Yes

    -

    -

    4000

    entryUUID

    -

    Yes

    -

    -

    -

    4000

    id2children

    Non-configurable internal index

    id2subtree

    Non-configurable internal index

    objectClass

    -

    Yes

    -

    -

    -

    4000

  2. Verify that replication is enabled:

    $ dsconfig \
 get-synchronization-provider-prop \
 --provider-name "Multimaster Synchronization" \
 --property enabled \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt
Property : Value(s)
---------:---------
enabled  : true

    If replication should be enabled but is not, use dsconfig set-synchronization-provider-prop --set enabled:true to enable it.

  3. Let the server replicate the base DN of the new backend:

    $ dsconfig \
 create-replication-domain \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --provider-name "Multimaster Synchronization" \
 --domain-name dc=example,dc=org \
 --type generic \
 --set enabled:true \
 --set base-dn:dc=example,dc=org \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

  4. If you have existing data for the backend, follow the appropriate procedure to initialize replication.

    For details, see Manual Initialization.

If you must temporarily disable replication for the backend, take care to avoid losing changes. For details, see Disable Replication.

Import and Export

The following procedures demonstrate how to import and export LDIF data.

For details on creating custom sample LDIF to import, refer to Generate test data.

Import LDIF

If you are initializing replicas by importing LDIF, refer to Initialize from LDIF for context.

  • Importing from LDIF overwrites all data in the target backend with entries from the LDIF data.

  • If the LDIF was exported from another server, it may contain pre-encoded passwords. As long as DS supports the password storage schemes used to encode the passwords, you can enable the storage schemes in the configuration to migrate existing passwords into DS. For details, see Password Storage.

    By default, password policies do not allow you to use pre-encoded passwords. You can change this behavior by changing the default password policy configuration property, allow-pre-encoded-passwords.

    The DS import process does not warn you about passwords that use disabled password storage schemes. Instead, search the LDIF to find all the password storage schemes in use, and make sure all the schemes are enabled in the server configuration. Users whose passwords are stored with a disabled scheme cannot bind successfully.

  • LDIF from another server can include passwords encrypted with a reversible storage scheme, such as AES or Blowfish. To decrypt the passwords, the server must use the same deployment key as the server that encrypted the passwords.

Perform either of the following steps to import dc=example,dc=com data into the dsEvaluation backend:

  • To import offline, shut down the server before you run the import-ldif command:

    $ stop-ds
$ import-ldif \
 --offline \
 --backendId dsEvaluation \
 --includeBranch dc=example,dc=com \
 --ldifFile example.ldif

  • To import online, schedule a task:

    $ start-ds
$ import-ldif \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backendId dsEvaluation \
 --includeBranch dc=example,dc=com \
 --ldifFile example.ldif \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin

    You can schedule the import task to start at a particular time using the --start option.

Export LDIF

Perform either of the following steps to export dc=example,dc=com data from the dsEvaluation backend:

  • To export offline, shut down the server before you run the export-ldif command:

    $ stop-ds
$ export-ldif \
 --offline \
 --backendId dsEvaluation \
 --includeBranch dc=example,dc=com \
 --ldifFile backup.ldif

  • To export online, schedule a task:

    $ export-ldif \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backendId dsEvaluation \
 --includeBranch dc=example,dc=com \
 --ldifFile backup.ldif \
 --start 0 \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin

    The --start 0 option tells the directory server to start the export task immediately.

    You can specify a time for the task to start using the format yyyymmddHHMMSS. For example, 20250101043000 specifies a start time of 4:30 AM on January 1, 2025.

    If the server is not running at the time specified, it attempts to perform the task after it restarts.

Split Data

Splitting data impacts replication, and can require that you interrupt the service.

On one set of replicas, create the exampleData backend to hold all dc=example,dc=com data except ou=People,dc=example,dc=com:

# Create a backend for the data not under the subordinate base DN:
$ dsconfig \
 create-backend \
 --backend-name exampleData \
 --type je \
 --set enabled:true \
 --set base-dn:dc=example,dc=com \
 --offline \
 --no-prompt

# Let the server replicate the base DN of the new backend:
$ dsconfig \
 create-replication-domain \
 --provider-name "Multimaster Synchronization" \
 --domain-name dc=example,dc=com \
 --type generic \
 --set enabled:true \
 --set base-dn:dc=example,dc=com \
 --offline \
 --no-prompt

# Import data not under the subordinate base DN:
$ import-ldif \
 --backendId exampleData \
 --excludeBranch ou=People,dc=example,dc=com \
 --ldifFile example.ldif \
 --offline

On another set of replicas, create a peopleData backend to hold ou=People,dc=example,dc=com data:

# Create a backend for the data under the subordinate base DN:
$ dsconfig \
 create-backend \
 --backend-name peopleData \
 --type je \
 --set enabled:true \
 --set base-dn:ou=People,dc=example,dc=com \
 --offline \
 --no-prompt

# Let the server replicate the base DN of the new backend:
$ dsconfig \
 create-replication-domain \
 --provider-name "Multimaster Synchronization" \
 --domain-name ou=People,dc=example,dc=com \
 --type generic \
 --set enabled:true \
 --set base-dn:ou=People,dc=example,dc=com \
 --offline \
 --no-prompt

# Import data under the subordinate base DN:
$ import-ldif \
 --backendId peopleData \
 --includeBranch ou=People,dc=example,dc=com \
 --ldifFile example.ldif \
 --offline

After completing the data import process, start the replicas.

To split an existing backend, follow these high-level steps:

  1. Stop the affected replica.

  2. Create the new backend.

  3. Export the data for the backend.

  4. Import the data under the subordinate base DN into the new backend, using the --includeBranch option.

  5. Delete all data under the subordinate base DN from the old backend.

  6. Start the affected replica.

Disk Space Thresholds

Directory data growth depends on applications that use the directory. When directory applications add more data than they delete, the database backend grows until it fills the available disk space. The system can end up in an unrecoverable state if no disk space is available.

Database backends therefore have advanced properties, disk-low-threshold and disk-full-threshold. When available disk space falls below disk-low-threshold, the directory server only allows updates from users and applications that have the bypass-lockdown privilege. When available space falls below disk-full-threshold, the directory server stops allowing updates, instead returning an UNWILLING_TO_PERFORM error to each update request.

If growth across the directory service tends to happen quickly, set the thresholds higher than the defaults to allow more time to react when growth threatens to fill the disk. The following example sets disk-low-threshold to 10 GB disk-full-threshold to 5 GB for the dsEvaluation backend:

$ dsconfig \
 set-backend-prop \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name dsEvaluation \
 --set "disk-low-threshold:10 GB" \
 --set "disk-full-threshold:5 GB" \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

The properties disk-low-threshold, and disk-full-threshold are listed as advanced properties.

To examine their values with the dsconfig command, use the --advanced option:

$ dsconfig \
 get-backend-prop \
 --advanced \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name dsEvaluation \
 --property disk-low-threshold \
 --property disk-full-threshold \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

Property            : Value(s)
--------------------:---------
disk-full-threshold : 5 gb
disk-low-threshold  : 10 gb

Entry Expiration

If the directory service creates many entries that expire and should be deleted, you could find the entries with a time-based search and then delete them individually. That approach uses replication to delete the entries in all replicas. It has the disadvantage of generating potentially large amounts of replication traffic.

Entry expiration lets you configure the backend database to delete the entries as they expire. This approach deletes expired entries at the backend database level, without generating replication traffic. AM uses this approach when relying on DS to delete expired token entries, as demonstrated in Install DS for AM CTS.

Backend indexes for generalized time (timestamp) attributes have these properties to configure automated, optimized entry expiration and removal:

Configure this capability by performing the following steps:

  1. Prepare an ordering index for a generalized time (timestamp) attribute on entries that expire.

    For details, see Configure Indexes and Active Accounts.

  2. Using the dsconfig set-backend-index-prop command, set ttl-enabled on the index to true, and set ttl-age on the index to the desired entry lifetime duration.

  3. Optionally enable the access log to record messages when the server deletes an expired entry.

    Using the dsconfig set-log-publisher-prop command, set suppress-internal-operations:false for the access log publisher. Note that this causes the server to log messages for all internal operations.

    When the server deletes an expired entry, it logs a message with "additionalItems":{"ttl": null} in the response.

Once you configure and build the index, the backend can delete expired entries. At intervals of 10 seconds, the backend automatically deletes entries whose timestamps on the attribute are older than the specified lifetime. Entries that expire in the interval between deletions are removed on the next round. Client applications should therefore check that entries have not expired, as it is possible for expired entries to remain available until the next round of deletions.

When using this capability, keep the following points in mind:

  • Entry expiration is per index. The time to live depends on the value of the indexed attribute and the ttl-age setting, and all matching entries are subject to TTL.

  • If multiple indexes' ttl-enabled and ttl-age properties are configured, as soon as one of the entry’s matching attributes exceeds the TTL, the entry is eligible for deletion.

  • The backend deletes the entries directly as an internal operation. The server only records the deletion when suppress-internal-operations: false for the access log publisher. Persistent searches do not return the deletion.

    Furthermore, this means that deletion is not replicated. To ensure expired entries are deleted on all replicas, use the same indexes with the same settings on all replicas.

  • When a backend deletes an expired entry, the effect is a subtree delete. In other words, if a parent entry expires, the parent entry and all the parent’s child entries are deleted.

    If you do not want parent entries to expire, index a generalized time attribute that is only present on its child entries.

  • The backend deletes expired entries atomically.

    If you update the TTL attribute to prevent deletion and the update succeeds, then TTL has effectively been reset.

  • Expiration fails when the index-entry-limit is exceeded. (For background information, see Index Entry Limits.)

    This only happens if the timestamp for the indexed attribute matches to the nearest millisecond on more than 4000 entries (for default settings). This corresponds to four million timestamp updates per second, which would be very difficult to reproduce in a real directory service.

    It is possible, however, to construct and import an LDIF file where more than 4000 entries have the same timestamp. Make sure not to reuse the same timestamp for thousands of entries when artificially creating entries that you intend to expire.

Add a Base DN to a Backend

The following example adds the base DN o=example to the dsEvaluation backend, and creates a replication domain configuration to replicate the data under the new base DN:

$ dsconfig \
 set-backend-prop \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name dsEvaluation \
 --add base-dn:o=example \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

$ dsconfig \
 get-backend-prop \
 --hostname localhost \
 --port 4444 \
 --bindDn uid=admin \
 --bindPassword password \
 --backend-name dsEvaluation \
 --property base-dn \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt
Property : Value(s)
---------:-------------------------------
base-dn  : "dc=example,dc=com", o=example

$ dsconfig \
 create-replication-domain \
 --provider-name "Multimaster Synchronization" \
 --domain-name o=example \
 --type generic \
 --set enabled:true \
 --set base-dn:o=example \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt

When you add a base DN to a backend, the base DN must not be subordinate to any other base DNs in the backend. As in the commands shown here, you can add the base DN o=example to a backend that already has a base DN dc=example,dc=com. You cannot, however, add o=example,dc=example,dc=com as a base DN, because that is a child of dc=example,dc=com.

Delete a Backend

When you delete a database backend with the dsconfig delete-backend command, the directory server does not actually remove the database files for these reasons:

  • A mistake could potentially cause lots of data to be lost.

  • Deleting a large database backend could cause severe service degradation due to a sudden increase in I/O load.

After you run the dsconfig delete-backend command, manually remove the database backend files, and remove replication domain configurations any base DNs you deleted by removing the backend.

If run the dsconfig delete-backend command by mistake, but have not yet deleted the actual files, recover the data by creating the backend again, and reconfiguring and rebuilding the indexes.

Copyright © 2010-2024 ForgeRock, all rights reserved.