Indexes in DS/OpenDJ

This book provides information on indexing in DS/OpenDJ and includes information on troubleshooting and known issues (with solutions).


How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)?

The purpose of this article is to provide assistance on troubleshooting issues with your indexes in DS/OpenDJ. You may be aware of issues with your indexes due to symptoms such as slow searches, poor performance, high CPU usage when searching and even long login times in AM/OpenAM when your user store is on DS/OpenDJ.

Troubleshooting indexes

Issues with indexes can be caused by one of the following:

General health check

You can use one of the following commands (depending on DS/OpenDJ version) to provide a general health check on all indexes to help you evaluate how well attributes are indexed:

  • DS / OpenDJ 3.x, you can use the following backendstat command; you must stop the server prior to running this command:
    $ ./backendstat show-index-status --backendID userRoot --baseDN dc=example,dc=com
  • OpenDJ 2.6.x, you can use the following dbtest command:
    $ ./dbtest list-index-status --backendID userRoot --baseDN dc=example,dc=com

These commands return a list of all indexes and includes information about the type of index, the associated database name, whether the index is valid and whether the index has exceeded its index entry limit.

Caution

These commands can be expensive and take a long time to complete, as they reads all indexes for all backends.

Degraded indexes

Degraded indexes occur when you have created or made changes to indexes and the index becomes corrupt due to missing data. You can check for degraded indexes using the verify-index command. This is described in How do I verify indexes in DS/OpenDJ (All versions) are correct?

Any degraded indexes must be rebuilt. You can rebuild specific indexes, all indexes or only degraded indexes as described in How do I rebuild indexes in DS/OpenDJ (All versions)?

Missing Indexes 

Missing indexes (unindexed) result in long search times (high etime values in the access log). This can even affect logging into AM/OpenAM if your user store is on DS/OpenDJ and an attribute involved in the login process is not indexed, for example, the LDAP SAML attribute.

Missing indexes need to be created and built.

See Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server for further information on the symptoms and solution.

Incomplete indexing

Incomplete indexing can also cause problems with slow searches, but sometimes it can be difficult to pinpoint which attribute is causing the issue if there are multiple attributes in the search filter. 

You can use the debugsearchindex attribute with the ldapsearch command to identify issues such as unindexed attributes and exceeded entry limits. The debugsearchindex attribute tells DS/OpenDJ to return an entry containing debug information that describes how the search operation would have been processed (rather than actually performing the search). See the debugsearchindex example below for further information.

Unnecessary indexes

Unused or rarely used indexes can also impact performance as DS/OpenDJ has to maintain entries for all indexes.

You can run a script (topfilters) to find out which indexes are used most often to help you identify underused indexes. You can then remove these indexes using the dsconfig delete-backend-index or delete-local-db-index command depending on which version of DS/OpenDJ you are using.

The source for the topfilters script can be found here: Githutb: OpenDJ Utilities - topfilters.

High index entry limits

Index entry limits are set to 4000 by default, which means that once the number of entries matching a given index exceeds this number, DS/OpenDJ stops maintaining the entries for that index as it is an inefficient use of resources. This default value should be sufficient for most indexes other than objectClass indexes.

Caution

Do not set the index-entry-limit to the number of entries in your backend.

If you have set your index's entry limits much higher than this for any of your indexes, you may find your performance is impacted as DS/OpenDJ has to maintain entries for very large indexes. You should try reducing the index entry limit back to the default value to see if this improves performance. 

Poorly defined search filters

Typically you should aim to have specific search filters in your searches to produce more accurate matches and also reduce the time taken to search.

For example, a search filter such as (objectClass=person) is very generalized and will produce a lot of matches. Instead, you should focus your search by adding other attributes to your search filter that would produce the matches you require. For example, you could use a filter similar to this to reduce the number of matches:

(&(mail=username@maildomain.net)(objectClass=person))

debugsearchindex example

You can use the information contained in the access logs to determine what you should include in your debugsearchindex. For example, the following sample access log snippets show high etimes for two different searches:

[19/Oct/2016:17:09:27 -0400] SEARCH conn=107918 op=1 msgID=2 base="ou=People,dc=example,dc=org" scope=one filter="(&(modifytimestamp>=20161019161001Z)(objectclass=person))" attrs="uid email" result=118 message="Client Disconnect" nentries=5 unindexed etime=16761

[19/Oct/2016:17:16:42 -0400] SEARCH conn=108320 op=1 msgID=2 base="ou=People,dc=example,dc=org" scope=one filter="(&(iscurrent=1)(objectclass=person))" attrs="uid email" result=118 message="Client Disconnect" nentries=9 unindexed etime=46961

You can then construct searches using this information to show which indexes are being used to help you identify incomplete or missing indexes. For example:

$ ./ldapsearch --hostname ds1.example.com --port 1389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN ou=People,dc=example,dc=org --searchScope one "(&(modifytimestamp>=20161019161001Z)(objectclass=person))" debugsearchindex

dn: cn=debugsearch
debugsearchindex: filter=(&(objectClass=person)[INDEX:objectClass.equality][LIMIT-EXCEEDED](modifyTimestamp>=20161019161001Z)[NOT-INDEXED])[NOT-INDEXED] scope=one[LIMIT-EXCEEDED:24482] final=[NOT-INDEXED]
$ ./ldapsearch --hostname ds1.example.com --port 1389 --bindDN "cn=Directory Manager" --bindPassword password --baseDN ou=People,dc=example,dc=org --searchScope one "(&(iscurrent=1)(objectclass=person))" debugsearchindex 

dn: cn=debugsearch
debugsearchindex: filter=(&(objectClass=person)[INDEX:objectClass.equality][LIMIT-EXCEEDED](iscurrent=1)[NOT-INDEXED])[NOT-INDEXED] scope=one[LIMIT-EXCEEDED:26021] final=[NOT-INDEXED]

See Administration Guide › Indexing Attribute Values › About debugsearchindex Values for further information on the results obtained from the debugsearchindex option.

See Also

Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

How do I verify indexes in DS/OpenDJ (All versions) are correct?

How do I rebuild indexes in DS/OpenDJ (All versions)?

How do I collect data for troubleshooting high CPU utilization on DS/OpenDJ (All versions) servers?

Administration Guide › Rebuilding Indexes

Administration Guide › Indexing Attribute Values

Administration Guide › Indexing Attribute Values › What To Index

Reference › Tools Reference

Configuration Reference

Related Training

ForgeRock Directory Services Core Concepts

Related Issue Tracker IDs

N/A


How do I know what index types are needed for search filters in DS/OpenDJ (All versions)?

The purpose of this article is to provide information on what index types are needed for search filters in DS/OpenDJ.

Index types needed for search filters

A search with single-level or subtree scope will need indexes. The index types required depend on all the terms in the search filter and are illustrated in the following table (the uid attribute is used as an example):

Filter Term Index Required
(uid=*) uid presence
(uid=m*) uid substring
(uid=*m) uid substring
(uid=*m*) uid substring
(uid=m*m) uid substring
(uid>=10) uid ordering
(uid<=10) uid ordering
(uid=m) uid equality
(uid~=doe) uid approximate

Further information about the different index types, with examples, is provided in Administration Guide › Indexing Attribute Values › Index Types and Their Functions.

See Also

Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)?

Administration Guide › Indexing Attribute Values › Index Types and Their Functions

Related Training

ForgeRock Directory Services Core Concepts

Related Issue Tracker IDs

N/A


How do I rebuild indexes in DS/OpenDJ (All versions)?

The purpose of this article is to provide information on rebuilding indexes in DS/OpenDJ. You need to rebuild indexes if you make configuration changes (adding, modifying or deleting indexes or index values) or if you experience degraded indexes.

Rebuilding indexes

Note

For OpenDJ 2.6.3 and earlier, you must stop the server and rebuild indexes offline, otherwise you will likely encounter known issue: OPENDJ-49 (Replication replay does not take into consideration the server/backend's writability mode.)

You can rebuild indexes using either the command line or the Control Panel (removed in DS 6):

  • command line: enter the following command to rebuild a specific index:
  • $ ./rebuild-index --port 4444 --hostname ds1.example.com --bindDN "cn=Directory Manager" --bindPassword password --baseDN "dc=example,dc=com" --index [indexname] --start 0
    You only need to specify a start time if you are rebuilding an index while the server is online; you can use --start 0 to rebuild an index immediately.
  • command line: enter the following command to rebuild all indexes:
    $ ./rebuild-index --port 4444 --hostname ds1.example.com --bindDN "cn=Directory Manager" --bindPassword password --baseDN "dc=example,dc=com" --rebuildAll
  • command line: enter the following command to rebuild all degraded indexes:
    $ ./rebuild-index --port 4444 --hostname ds1.example.com --bindDN "cn=Directory Manager" --bindPassword password --baseDN "dc=example,dc=com" --rebuildDegraded
  • Control Panel: navigate to: Indexes > Rebuild Indexes and select the indexes you want to rebuild. Indexes that require rebuilding (they are degraded or have had configuration changes) are indicated by (*).

You can verify an index after it has been rebuilt to check that it is no longer degraded. See How do I verify indexes in DS/OpenDJ (All versions) are correct? for further information.

See Also

How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)?

Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

How do I know what index types are needed for search filters in DS/OpenDJ (All versions)?

Indexes in DS/OpenDJ

Administration Guide › Rebuilding Indexes

Reference › rebuild-index

Administration Guide › Verifying Indexes

Related Training

ForgeRock Directory Services Core Concepts

Related Issue Tracker IDs

OPENDJ-49 (Replication replay does not take into consideration the server/backend's writability mode.)

OPENDJ-1266 (State index is not updated when an index is deleted)


How do I verify indexes in DS/OpenDJ (All versions) are correct?

The purpose of this article is to provide some guidance on checking DS/OpenDJ indexes are correct.

Index Verification

Indexes in DS/OpenDJ can be incorrect in two different ways:

  1. An index contains a key that refers to an entry that no longer exists or no longer contains the indexed value.
  2. An index should contain a reference to an entry for a particular key, but does not.

Both cases should not occur in general usage.

To detect the errors, the verify-index tool is provided with DS/OpenDJ. It needs to be run with different options to detect both kinds of index problems. As of OpenDJ 3, you must stop the server before verifying indexes.

Note

For an attribute index, the index name is simply an attribute name. Multiple indexes may be verified for completeness or all indexes if no index name is specified.

Verify Case 1

Run the following verify-index command to step through all the entries in the backend and verify that all indexes are correct:

$ ./verify-index --baseDN [baseDN] --countErrors

replacing [baseDN] with appropriate values.

Verify Case 2

Run the following verify-index command to step through all the keys in the index and verify the values for each key correctly match entries in the backend:

$ ./verify-index --baseDN [baseDN] --index [indexname] --clean

replacing [baseDN] and [indexname] with appropriate values.

Verify Case 3

Run the following verify-index command to step through all the entries in the backend and verify that each entry is referenced in the index if appropriate:

$ ./verify-index --baseDN [baseDN] --index [indexname]

replacing [baseDN] and [indexname] with appropriate values.

Examples

The objectClass index on this DS/OpenDJ server is completely valid:

$ ./verify-index --baseDN "dc=example,dc=com" --countErrors
[20/11/2017:10:35:54 -0700] category=BACKEND seq=1 severity=FINE msg=JE backend 'userRoot' does not specify the number of cleaner threads: defaulting to 8 threads
[20/11/2017:10:35:54 -0700] category=BACKEND seq=2 severity=FINE msg=JE backend 'userRoot' does not specify the number of lock tables: defaulting to 17
[20/11/2017:10:35:57 -0700] category=BACKEND seq=35 severity=INFO msg=Checked 10002 entries and found 0 error(s) in 2 seconds (average rate 3965.9/sec)
[20/11/2017:10:35:57 -0700] category=BACKEND seq=36 severity=FINE msg=Statistics for records that have exceeded the entry limit:
[20/11/2017:10:35:57 -0700] category=BACKEND seq=37 severity=FINE msg=File /dc=com,dc=example/mail.caseIgnoreIA5SubstringsMatch:6 has 15 such record(s) min=10000 max=10000 median=10000
[20/11/2017:10:35:57 -0700] category=BACKEND seq=38 severity=FINE msg=File /dc=com,dc=example/objectClass.objectIdentifierMatch has 4 such record(s) min=10000 max=10002 median=10000
$ ./verify-index --baseDN "dc=example,dc=com" --index objectClass --clean
[20/Oct/2016:11:44:42 +0100] category=BACKEND severity=INFORMATION msgID=9437595 msg=Local DB backend userRoot does not specify the number of lock tables: defaulting to 97
[20/Oct/2016:11:44:42 +0100] category=BACKEND severity=INFORMATION msgID=9437594 msg=Local DB backend userRoot does not specify the number of cleaner threads: defaulting to 24 threads
[20/Oct/2016:11:44:42 +0100] category=JEB severity=NOTICE msgID=8847461 msg=Checked 6 records and found 0 error(s) in 0 seconds (average rate 200.0/sec)
[20/Oct/2016:11:44:42 +0100] category=JEB severity=INFORMATION msgID=8388710 msg=Number of records referencing more than one entry: 4
[20/Oct/2016:11:44:42 +0100] category=JEB severity=INFORMATION msgID=8388711 msg=Number of records that exceed the entry limit: 4
[20/Oct/2016:11:44:42 +0100] category=JEB severity=INFORMATION msgID=8388712 msg=Average number of entries referenced is 0.33/record
[20/Oct/2016:11:44:42 +0100] category=JEB severity=INFORMATION msgID=8388713 msg=Maximum number of entries referenced by any record is 1
$ ./verify-index --baseDN "dc=example,dc=com" --index objectClass
[20/Oct/2016:11:43:07 +0100] category=BACKEND severity=INFORMATION msgID=9437595 msg=Local DB backend userRoot does not specify the number of lock tables: defaulting to 97
[20/Oct/2016:11:43:07 +0100] category=BACKEND severity=INFORMATION msgID=9437594 msg=Local DB backend userRoot does not specify the number of cleaner threads: defaulting to 24 threads
[20/Oct/2016:11:43:09 +0100] category=JEB severity=NOTICE msgID=8847466 msg=Checked 20002 entries and found 0 error(s) in 1 seconds (average rate 11984.4/sec)
[20/Oct/2016:11:43:09 +0100] category=JEB severity=INFORMATION msgID=8388715 msg=Statistics for records that have exceeded the entry limit:
[20/Oct/201614:11:43:09 +0100] category=JEB severity=INFORMATION msgID=8388716 msg=File dc_example_dc_com_objectClass.equality has 4 such record(s) min=20000 max=20002 median=20000

See Also

How do I rebuild indexes in DS/OpenDJ (All versions)?

Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)?

How do I know what index types are needed for search filters in DS/OpenDJ (All versions)?

Administration Guide › Indexing Attribute Values › Verifying Indexes

Reference › Tools Reference › verify-index

Administration Guide › Indexing Attribute Values

Related Training

ForgeRock Directory Services Core Concepts

Related Issue Tracker IDs

OPENDJ-1266 (State index is not updated when an index is deleted)


Known Issues


Unindexed searches causing slow searches and poor performance on DS/OpenDJ (All versions) server

The purpose of this article is to provide assistance if you have unindexed searches that are causing slow searches and poor performance on the DS/OpenDJ server. Other symptoms may include high CPU usage when searching.

Symptoms

An entry similar to the following is shown in the access log (located in the /path/to/ds/logs directory where DS/OpenDJ is installed):

[21/Jan/2015:11:37:15 +0000] SEARCH REQ conn=159078 op=1403 msgID=154 base="ou=groups,dc=amusers" scope=wholeSubtree filter="(memberUid=JDoe)" attrs="1.1"
[21/Jan/2015:11:37:27 +0000] SEARCH RES conn=159078 op=1403 msgID=154 result=0 nentries=0 unindexed etime=11999

where the search is shown as unindexed and has a high etime value (etime values should typically be 2 or 3).

Alternatively, an entry may show with a high etime without stating it is unindexed:

[23/Feb/2015:10:55:58 -0000] SEARCH RES conn=2220857 op=1335 msgID=1136 result=0 nentries=1 etime=1187

Unindexed only appears in the access log if no indexes were used to process the search, but the high etime still suggests a partially unindexed search.

Recent Changes

N/A

Causes

Indexes improve search performance based on search filters. DS/OpenDJ comes with a set of standard indexes for commonly used attributes, such as givenName, to improve searches that include this attribute. When an attribute is included in a search that has not been indexed, the search takes a lot longer (reflected in the high etime value). 

Solution

This issue can be resolved by creating and building an index for the affected attribute. In the first example log entry, the affected attribute is memberUid, so you would need to create and build an index for this attribute.

If a search filter includes multiple attributes and you are unsure which attribute is unindexed, you can use the debugsearchindex option with the ldapsearch command to find out how DS/OpenDJ would process the search (rather than actually performing the search) and identify any unindexed attributes. See the debugsearchindex example in How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)? for further information.

See Administration Guide › Indexing Attribute Values › Configuring a Standard IndexHow do I know what index types are needed for search filters in DS/OpenDJ (All versions)?How do I rebuild indexes in DS/OpenDJ (All versions)? and Administration Guide › Indexing Attribute Values › Rebuilding Indexes for further information on creating and building indexes.

Caution

You should only create and build indexes for attributes that are repeatedly causing high etimes and are included in your search filters as too many indexes can actually be detrimental to performance, especially if they are unused.

See Also

How do I troubleshoot issues with my indexes in DS/OpenDJ (All versions)?

How do I rebuild indexes in DS/OpenDJ (All versions)?

How do I know what index types are needed for search filters in DS/OpenDJ (All versions)?

Administration Guide › Indexing Attribute Values › Configuring a Standard Index

Administration Guide › Indexing Attribute Values

Administration Guide › Indexing Attribute Values › What To Index

Administration Guide › Indexing Attribute Values › Rebuilding Indexes

idmdude - OpenDJ Indexes Explained

Related Training

ForgeRock Directory Services Core Concepts

Related Issue Tracker IDs

N/A


Out Of Memory Error when installing OpenDJ 3, or using import-ldif, rebuild-index or dsreplication commands

The purpose of this article is to provide assistance if you encounter an "Exception in thread "main" java.lang.OutOfMemoryError at sun.misc.Unsafe.allocateMemory(Native Method)" error. This error can occur when you use the setup command to install or upgrade, or you use the import-ldif or rebuild-index commands. Similarly, you might see an "OutOfMemoryError (Unsafe.java" error when using the dsreplication command.

Symptoms

You will see one of the following errors in your errors log depending on which command you are using:

  • setup (this includes an import-ldif command that's used during the setup process):
    [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=73 severity=INFO msg=import-ldif out log: [12/09/2016:08:39:51 +0100] category=PLUGGABLE seq=11 severity=INFO msg=Import LDIF environment close took 0 seconds
    [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=74 severity=INFO msg=import-ldif out log: [12/09/2016:08:39:51 +0100] category=PLUGGABLE seq=12 severity=INFO msg=Flushing data to disk
    [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=75 severity=WARNING msg=import-ldif error log: Exception in thread "main" java.lang.OutOfMemoryError
    [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=76 severity=WARNING msg=import-ldif error log:  at sun.misc.Unsafe.allocateMemory(Native Method)
    [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=77 severity=WARNING msg=import-ldif error log:  at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool$OffHeapBuffer.<init>(OnDiskMergeImporter.java:2858)
    [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=78 severity=WARNING msg=import-ldif error log:  at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool.<init>(OnDiskMergeImporter.java:2780)
    [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=79 severity=WARNING msg=import-ldif error log:  at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.importLDIF(OnDiskMergeImporter.java:203)
    [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=80 severity=WARNING msg=import-ldif error log:  at org.opends.server.backends.pluggable.BackendImpl.importLDIF(BackendImpl.java:689)
    ...
    
  • import-ldif:
    Exception in thread "main" java.lang.OutOfMemoryError 
       at sun.misc.Unsafe.allocateMemory(Native Method) 
       at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool$OffHeapBuffer.<init>(OnDiskMergeImporter.java:2858) 
       at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool.<init>(OnDiskMergeImporter.java:2780) 
       at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.importLDIF(OnDiskMergeImporter.java:203) 
       at org.opends.server.backends.pluggable.BackendImpl.importLDIF(BackendImpl.java:689) 
       at org.opends.server.tools.ImportLDIF.processLocal(ImportLDIF.java:1092) 
       at org.opends.server.tools.tasks.TaskTool.process(TaskTool.java:362) 
       at org.opends.server.tools.ImportLDIF.process(ImportLDIF.java:292) 
       at org.opends.server.tools.ImportLDIF.mainImportLDIF(ImportLDIF.java:147) 
       at org.opends.server.tools.ImportLDIF.main(ImportLDIF.java:110)
    ...
    
  • rebuild-index:
    Exception in thread "main" java.lang.OutOfMemoryError 
       at sun.misc.Unsafe.allocateMemory(Native Method) 
       at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool$OffHeapBuffer.<init>(OnDiskMergeImporter.java:2858) 
       at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool.<init>(OnDiskMergeImporter.java:2780) 
       at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.rebuildIndex(OnDiskMergeImporter.java:312) 
       at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.rebuildIndex(OnDiskMergeImporter.java:275) 
       at org.opends.server.backends.pluggable.BackendImpl.rebuildBackend(BackendImpl.java:806) 
       at org.opends.server.tools.RebuildIndex.rebuildIndex(RebuildIndex.java:559) 
       at org.opends.server.tools.RebuildIndex.processLocal(RebuildIndex.java:321) 
       at org.opends.server.tools.tasks.TaskTool.process(TaskTool.java:362) 
       at org.opends.server.tools.RebuildIndex.process(RebuildIndex.java:228) 
       at org.opends.server.tools.RebuildIndex.mainRebuildIndex(RebuildIndex.java:138) 
       at org.opends.server.tools.RebuildIndex.main(RebuildIndex.java:110) 
    ...
    
  • dsreplication:
    [17/Sep/2016:08:39:51 +0100] category=org.opends.server.api.DirectoryThread severity=ERROR msgID=org.opends.messages.core.140 msg=An uncaught exception during processing for thread Replica DS(18687) listener for domain "dc=forgerock,dc=com" has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError (Unsafe.java:-2 OnDiskMergeImporter.java:2858 OnDiskMergeImporter.java:2780 OnDiskMergeImporter.java:203 BackendImpl.java:689 LDAPReplicationDomain.java:3565 ReplicationDomain.java:2275 ReplicationDomain.java:778 ReplicationDomain.java:106 ReplicationDomain.java:2965 Thread.java:744)
    [17/Sep/2016:08:39:51 +0100] category=CORE severity=NOTICE msgID=org.opends.messages.core.139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Replica DS(18687) listener for domain "dc=forgerock,dc=com" has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError (Unsafe.java:-2 OnDiskMergeImporter.java:2858 OnDiskMergeImporter.java:2780 OnDiskMergeImporter.java:203 BackendImpl.java:689 LDAPReplicationDomain.java:3565 ReplicationDomain.java:2275 ReplicationDomain.java:778 ReplicationDomain.java:106 ReplicationDomain.java:2965 Thread.java:744)
    ...
    

Recent Changes

Upgraded to, or installed OpenDJ 3.0.

Causes

Changes were made in OpenDJ 3.0.0 tools that introduced a new off-heap memory management mechanism, which mean that certain commands now request memory outside of the Java® heap using the sun.misc.Unsafe.allocateMemory() method. 

The number of allocations depend on the number of threads and the number of attribute indexes. OpenDJ calculates how many threads it uses based on the number of CPUs available to the system. It is likely this error occurs because the number of allocations being done when you are using the default number of threads exceeds the amount of memory a 32-bit JVM can allocate.

Solution

This issue can be resolved by upgrading to OpenDJ 3.5 or later which has improved/fixed memory management for imports; you can download this from BackStage.

Workaround

You can workaround this issue by switching to a 64-bit JVM as follows to allow more memory to be allocated:

  1. Add the -d64 option to the start-ds.java-args property in the java.properties file (located in the /path/to/opendj/config directory), for example:
    start-ds.java-args=-server -Xms2g -Xmx2g -d64
    
  2. Run the bin/dsjavaproperties command to apply the changes you have made to the java.properties file:
    $ ./dsjavaproperties
  3. Restart the OpenDJ server.
Note

If your heap size is less than 32GB, you should also specify CompressedOops as detailed in How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache?

If your JVM does not support 64-bit, you can try one of the following approaches instead:

  • Limit the number of CPUs being used by these commands as this will reduce the amount of memory required. This is a server level change, but for import-ldif you can limit the number of threads being used by adding the --threadCount option to your command. For example, add the following to use 8 threads:
    --threadCount 8
  • Increase the JVM heap size. See How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache? for further information.
  • Run ./setup without creating the baseDN entry (that is, exclude the --addBaseEntry or -a option) and then run an import-ldif command afterwards to create it.

See Also

OpenDJ Release Notes › OpenDJ Fixes, Limitations, and Known Issues › Key Fixes in 3.5.0

How do I tune DS/OpenDJ (All versions) process sizes: JVM heap and database cache?

How do I ensure DS/OpenDJ (All versions) uses the Java settings from java.properties file when DS/OpenDJ is started?

Best practice for JVM Tuning

OpenDJ Administration Guide › Managing Directory Data › Importing and Exporting Data

How do I collect JVM data for troubleshooting DS/OpenDJ (All versions)?

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-3212 (When we are upgrading Opendj from 2.4.4 to 3.0.0 version, java.lang.OutOfMemoryError occured)

OPENDJ-2721 (JE is using all the available heap memory during import.)


Embedded OpenDJ tools fail in OpenAM 13.5.x with an Unable to load class or Could not find or load main class error

The purpose of this article is to provide assistance if you receive an "Unable to load class org.opends.server.backends.jeb.JEBackend" or "Could not find or load main class org.opends.server.tools" error when attempting to use embedded OpenDJ tools (such as dsreplication, import-ldif or ldapsearch) in OpenAM.

Symptoms

The following error is shown when embedded OpenDJ tools such as import-ldif fail in OpenAM:

[25/08/2016:17:19:43 +0000] category=TOOLS seq=0 severity=SEVERE msg=Unable to load class org.opends.server.backends.jeb.JEBackend referenced in configuration entry ds-cfg-backend-id=userRoot,cn=Backends,cn=config for use as a Directory Server backend: ClassNotFoundException(org.opends.server.backends.jeb.JEBackend)

You may also see errors similar to the following when embedded OpenDJ tools such as dsreplication or ldapsearch fail if you have upgraded to OpenAM 13.5.x:

Error: Could not find or load main class org.opends.server.tools.dsreplication.ReplicationCliMain

Error: Could not find or load main class org.opends.server.tools.LDAPSearch

Recent Changes

Upgraded to, or installed OpenAM 13.5.x.

Causes

The necessary JE libraries are missing from the embedded OpenDJ installation in OpenAM 13.5.x, which cause the embedded OpenDJ tools to fail with the "Unable to load class" error.

  • OpenAM 13.5 - missing both the opendj-je-backend.jar and je.jar files.
  • OpenAM 13.5.1 and 13.5.2 - missing the opendj-je-backend.jar file only. The je.jar file has been replaced with a forgerock-je.jar file, which is included in the install.

The opendj.jar file is missing from the embedded OpenDJ installation in OpenAM 13.5.x if you have upgraded to OpenAM (instead of doing a fresh install); this can cause the "Could not find or load main class" errors when attempting to use the embedded OpenDJ tools.

Solution

These issues can be resolved as follows depending on which error(s) you are encountering and your version:

"Unable to load class" error (OpenAM 13.5.1 and 13.5.2):

  1. Copy the opendj-je-backend.jar file from the /path/to/tomcat/webapps/openam/WEB-INF/lib/ directory where OpenAM is deployed to the $HOME/[openam_instance]/opends/lib directory.

"Unable to load class" error (OpenAM 13.5):

  1. Copy the opendj-je-backend.jar and je-5.0.104.jar files from the /path/to/tomcat/webapps/openam/WEB-INF/lib/ directory where OpenAM is deployed to the $HOME/[openam_instance]/opends/lib directory.
  2. Rename the je-5.0.104.jar to je.jar.

"Could not find or load main class" error (OpenAM 13.5.x)

  1. Navigate to the following directory:
    /path/to/tomcat/webapps/openam/WEB-INF/template/opendj
  2. Unzip the opendj.zip file:
    $ unzip opendj.zip
    
  3. Copy the opendj.jar file from the exploded lib directory to the embedded OpenDJ lib directory:
    $HOME/[am_instance]/opends/opends/lib/
    
Note

You can also prepare the original OpenAM 13.5.x war file to include these files to ensure any future installs are not affected by this issue. You should add the required files to the WEB-INF/template/opendj/opendj.zip and recompile the OpenAM 13.5.x war file.

See Also

N/A

Related Training

N/A

Related Issue Tracker IDs

OPENAM-9715 (Embedded OpenDJ backup and verify-index fails in OpenAM 13.5.0)

OPENAM-9593 (Embedded OpenDJ installation is missing JE libraries)


Copyright and TrademarksCopyright © 2018 ForgeRock, all rights reserved.

This content has been optimized for printing.

Loading...