JE database backend growing rapidly in OpenDJ 3.0, 3.5 and 3.5.1

Last updated Jan 5, 2021

The purpose of this article is to provide assistance if a database backend (for example, userRoot) starts growing rapidly in OpenDJ 3.0, 3.5 and 3.5.1 for no apparent reason. This issue can cause you to run out of disk space if left unchecked. The embedded OpenDJ can also be affected in OpenAM 13.x.

Archived

This article has been archived and is no longer maintained by ForgeRock.

Symptoms

A database backend grows rapidly for no reason. Restarting the server may restore the database backend to its usual size.

You may see signs in the je.info.0 file (located in the /path/to/opendj/db/backend_name directory) that the cleaning process is not working effectively for the affected backend. Things to look for are as follows, where the backend in these examples is userRoot:

lnSizeCorrectionFactor drops below 0.5, for example: 2017-01-10 10:37:22.477 GMT INFO [/path/to/opendj/db/userRoot] Chose lowest utilized file for cleaning. fileChosen: 0xf044 lnSizeCorrectionFactor: 0.341327 totalUtilization: 49 bestFileUtilization: 10 isProbe: false
Files are not being deleted (fileDeleted=false), for example: 2017-01-10 10:37:21.429 GMT INFO [/path/to/opendj/db/userRoot] CleanerRun 788 ends on file 0xf044 probe=false invokedFromDaemon=true finished=true fileDeleted=false nEntriesRead=20455 nINsObsolete=4304 nINsCleaned=1314 nINsDead=0 nINsMigrated=1314 nBINDeltasObsolete=7925 nBINDeltasCleaned=1794 nBINDeltasDead=0 nBINDeltasMigrated=1794 nLNsObsolete=1406 nLNsCleaned=3709 nLNsDead=0 nLNsMigrated=3709 nLNsMarked=0 nLNQueueHits=1412 nLNsLocked=0 logSummary=<CleanerLogSummary endFileNumAtLastAdjustment="0xf052" initialAdjustments="5" recentLNSizesAndCounts="Cor:172/1-Est:9571857/3694 Cor:140/1-Est:10254376/3164 Cor:124/1-Est:9360777/4179 Cor:5536/4-Est:9867834/6213 Cor:59139/26-Est:7818668/3255 Cor:214/1-Est:10019874/4069 Cor:1734/3-Est:10336372/4604 Cor:9155/5-Est:8702308/4023 Cor:156/1-Est:9266763/4814 Cor:69442/30-Est:7293631/3739 "> inSummary=<INSummary totalINCount="5618" totalINSize="11579684" totalBINDeltaCount="9719" totalBINDeltaSize="1645150" obsoleteINCount="4304" obsoleteINSize="9408393" obsoleteBINDeltaCount="7925" obsoleteBINDeltaSize="1409529"/> estFileSummary=<summary totalCount="20836" totalSize="99958921" totalINCount="15337" totalINSize="13224834" totalLNCount="5115" totalLNSize="86713267" maxLNSize="798458" obsoleteINCount="12229" obsoleteLNCount="1406" obsoleteLNSize="79419636" obsoleteLNSizeCounted="1376" getObsoleteSize="90043834" getObsoleteINSize="10544858" getObsoleteLNSize="79478156" getMaxObsoleteSize="97219601" getMaxObsoleteLNSize="86653923" getAvgObsoleteLNSizeNotCounted="1950.6903"/> recalcFileSummary=<summary totalCount="20836" totalSize="99958921" totalINCount="15337" totalINSize="13224834" totalLNCount="5115" totalLNSize="86713267" maxLNSize="0" obsoleteINCount="12229" obsoleteLNCount="1406" obsoleteLNSize="79489078" obsoleteLNSizeCounted="1406" getObsoleteSize="90054756" getObsoleteINSize="10544858" getObsoleteLNSize="79489078" getMaxObsoleteSize="90054756" getMaxObsoleteLNSize="79489078" getAvgObsoleteLNSizeNotCounted="NaN"/> lnSizeCorrection=0.33163622 newLnSizeCorrection=0.33163622 estimatedUtilization=10 correctedUtilization=10 recalcUtilization=10 correctionRejected=false

Checking space utilization

You can also check space utilization as follows:

Change to the OpenDJ directory:
- External OpenDJ: $ cd /path/to/opendj
- Embedded OpenDJ: $ cd $HOME/[openam_instance]/opends
Run the following command (replacing backend with the actual name of your backend): $ java -cp lib/je.jar:lib/OpenDJ.jar com.sleepycat.je.util.DbSpace -h db/backend The presence of 0 occupancy files indicates the cleaning process is not working as expected, for example, if you see an output such as: File Size (KB) % Used -------- --------- ------ 0xf044 97654 2 0xf045 97653 0 0xf046 97653 0 0xf047 97653 0 0xf048 97654 0 0xf049 97655 0

Recent Changes

N/A

Causes

An underlying bug in JE 5 is responsible for this issue. It causes the cleaning process to be adversely affected, which leaves the database full of log files that have a high percentage of deleted records. This bug is fixed in JE 6 and later.

A JE database backend is effectively an ordered sequence of append-only transaction log files with the jdb extension. Each one can grow up to a configurable maximum, where the default maximum size is 100MB (ds-cfg-db-log-file-max). Each log file contains a percentage of live records and (when data is modified) deleted records. These percentages are monitored, and when the amount of live data falls below 50% (this is also configurable), cleaner threads actively migrate the remaining live records to the end of the transaction log (the latest jdb file) and remove the log that's now full of deleted records.

Warning

You must not delete these jdb files yourself, else you will corrupt the database.

Solution

This issue can be resolved by upgrading to DS 5 and later, or OpenDJ 3.5.2; you can download these from BackStage.

These newer versions use JE 7, which is under an Apache™ License.

Workaround

A suggested workaround is to try setting the je.cleaner.adjustUtilization property to false. This setting causes the cleaner to assume that all records in a log are average size; this may occasionally result in under or over cleaning, but overall cleaning should be consistent and can prevent this issue from re-occurring.

For example, to set this for the userRoot database backend, you would use a dsconfig command such as:

$ ./dsconfig set-backend-prop --backend-name userRoot --set je-property:je.cleaner.adjustUtilization=false --hostname localhost--port 4444 --bindDN "cn=Directory Manager" --bindPassword password --no-prompt --trustAll

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-3428 (JE cleaner threads stop deleting files)

OPENDJ-2362 (Setting CLEANER_MIN_FILE_UTILIZATION instead of CLEANER_MIN_UTILIZATION)

OPENDJ-1554 (Backend in default config.ldif does not use the correct defaults)