Solutions
Archived

Out Of Memory Error when installing OpenDJ 3, or using import-ldif, rebuild-index or dsreplication commands

Last updated Jan 5, 2021

The purpose of this article is to provide assistance if you encounter an "Exception in thread "main" java.lang.OutOfMemoryError at sun.misc.Unsafe.allocateMemory(Native Method)" error. This error can occur when you use the setup command to install or upgrade, or you use the import-ldif or rebuild-index commands. Similarly, you might see an "OutOfMemoryError (Unsafe.java" error when using the dsreplication command.


Archived

This article has been archived and is no longer maintained by ForgeRock.

Symptoms

You will see one of the following errors in your errors log depending on which command you are using:

  • setup (this includes an import-ldif command that's used during the setup process): [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=73 severity=INFO msg=import-ldif out log: [12/09/2016:08:39:51 +0100] category=PLUGGABLE seq=11 severity=INFO msg=Import LDIF environment close took 0 seconds [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=74 severity=INFO msg=import-ldif out log: [12/09/2016:08:39:51 +0100] category=PLUGGABLE seq=12 severity=INFO msg=Flushing data to disk [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=75 severity=WARNING msg=import-ldif error log: Exception in thread "main" java.lang.OutOfMemoryError [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=76 severity=WARNING msg=import-ldif error log: at sun.misc.Unsafe.allocateMemory(Native Method) [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=77 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool$OffHeapBuffer.<init>(OnDiskMergeImporter.java:2858) [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=78 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool.<init>(OnDiskMergeImporter.java:2780) [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=79 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.importLDIF(OnDiskMergeImporter.java:203) [12/09/2016:08:39:51 +0100] category=QUICKSETUP seq=80 severity=WARNING msg=import-ldif error log: at org.opends.server.backends.pluggable.BackendImpl.importLDIF(BackendImpl.java:689) ...
  • import-ldif: Exception in thread "main" java.lang.OutOfMemoryError at sun.misc.Unsafe.allocateMemory(Native Method) at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool$OffHeapBuffer.<init>(OnDiskMergeImporter.java:2858) at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool.<init>(OnDiskMergeImporter.java:2780) at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.importLDIF(OnDiskMergeImporter.java:203) at org.opends.server.backends.pluggable.BackendImpl.importLDIF(BackendImpl.java:689) at org.opends.server.tools.ImportLDIF.processLocal(ImportLDIF.java:1092) at org.opends.server.tools.tasks.TaskTool.process(TaskTool.java:362) at org.opends.server.tools.ImportLDIF.process(ImportLDIF.java:292) at org.opends.server.tools.ImportLDIF.mainImportLDIF(ImportLDIF.java:147) at org.opends.server.tools.ImportLDIF.main(ImportLDIF.java:110) ...
  • rebuild-index: Exception in thread "main" java.lang.OutOfMemoryError at sun.misc.Unsafe.allocateMemory(Native Method) at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool$OffHeapBuffer.<init>(OnDiskMergeImporter.java:2858) at org.opends.server.backends.pluggable.OnDiskMergeImporter$BufferPool.<init>(OnDiskMergeImporter.java:2780) at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.rebuildIndex(OnDiskMergeImporter.java:312) at org.opends.server.backends.pluggable.OnDiskMergeImporter$StrategyImpl.rebuildIndex(OnDiskMergeImporter.java:275) at org.opends.server.backends.pluggable.BackendImpl.rebuildBackend(BackendImpl.java:806) at org.opends.server.tools.RebuildIndex.rebuildIndex(RebuildIndex.java:559) at org.opends.server.tools.RebuildIndex.processLocal(RebuildIndex.java:321) at org.opends.server.tools.tasks.TaskTool.process(TaskTool.java:362) at org.opends.server.tools.RebuildIndex.process(RebuildIndex.java:228) at org.opends.server.tools.RebuildIndex.mainRebuildIndex(RebuildIndex.java:138) at org.opends.server.tools.RebuildIndex.main(RebuildIndex.java:110) ...
  • dsreplication: [17/Sep/2016:08:39:51 +0100] category=org.opends.server.api.DirectoryThread severity=ERROR msgID=org.opends.messages.core.140 msg=An uncaught exception during processing for thread Replica DS(18687) listener for domain "dc=forgerock,dc=com" has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError (Unsafe.java:-2 OnDiskMergeImporter.java:2858 OnDiskMergeImporter.java:2780 OnDiskMergeImporter.java:203 BackendImpl.java:689 LDAPReplicationDomain.java:3565 ReplicationDomain.java:2275 ReplicationDomain.java:778 ReplicationDomain.java:106 ReplicationDomain.java:2965 Thread.java:744) [17/Sep/2016:08:39:51 +0100] category=CORE severity=NOTICE msgID=org.opends.messages.core.139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Replica DS(18687) listener for domain "dc=forgerock,dc=com" has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError (Unsafe.java:-2 OnDiskMergeImporter.java:2858 OnDiskMergeImporter.java:2780 OnDiskMergeImporter.java:203 BackendImpl.java:689 LDAPReplicationDomain.java:3565 ReplicationDomain.java:2275 ReplicationDomain.java:778 ReplicationDomain.java:106 ReplicationDomain.java:2965 Thread.java:744) ...

Recent Changes

Upgraded to, or installed OpenDJ 3.0.

Causes

Changes were made in OpenDJ 3.0.0 tools that introduced a new off-heap memory management mechanism, which mean that certain commands now request memory outside of the Java® heap using the sun.misc.Unsafe.allocateMemory() method. 

The number of allocations depend on the number of threads and the number of attribute indexes. OpenDJ calculates how many threads it uses based on the number of CPUs available to the system. It is likely this error occurs because the number of allocations being done when you are using the default number of threads exceeds the amount of memory a 32-bit JVM can allocate.

Solution

This issue can be resolved by upgrading to OpenDJ 3.5 or later which has improved/fixed memory management for imports; you can download this from BackStage.

Workaround

You can workaround this issue by switching to a 64-bit JVM as follows to allow more memory to be allocated:

  1. Add the -d64 option to the start-ds.java-args property in the java.properties file (located in the /path/to/opendj/config directory), for example: start-ds.java-args=-server -Xms2g -Xmx2g -d64
  2. Run the bin/dsjavaproperties command to apply the changes you have made to the java.properties file: $ ./dsjavaproperties
  3. Restart the OpenDJ server.
Note

If your heap size is less than 32GB, you should also specify CompressedOops as detailed in How do I tune DS (All versions) process sizes: JVM heap and database cache?

If your JVM does not support 64-bit, you can try one of the following approaches instead:

  • Limit the number of CPUs being used by these commands as this will reduce the amount of memory required. This is a server level change, but for import-ldif you can limit the number of threads being used by adding the --threadCount option to your command. For example, add the following to use 8 threads: --threadCount 8
  • Increase the JVM heap size. See How do I tune DS (All versions) process sizes: JVM heap and database cache? for further information.
  • Run ./setup without creating the baseDN entry (that is, exclude the --addBaseEntry or -a option) and then run an import-ldif command afterwards to create it.

See Also

OpenDJ Release Notes › OpenDJ Fixes, Limitations, and Known Issues › Key Fixes in 3.5.0

How do I tune DS (All versions) process sizes: JVM heap and database cache?

How do I ensure DS (All versions) uses the Java settings from java.properties file when starting?

Best practice for JVM Tuning with CMS GC

OpenDJ Administration Guide › Managing Directory Data › Importing and Exporting Data

How do I collect JVM data for troubleshooting DS (All versions)?

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-3212 (When we are upgrading Opendj from 2.4.4 to 3.0.0 version, java.lang.OutOfMemoryError occured)

OPENDJ-2721 (JE is using all the available heap memory during import.)


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.