Solutions
Archived

Upgrade to OpenAM 12.0.0 or 12.0.1 hangs in DirectoryContentUpgrader when configuration and CTS stores are external

Last updated Jan 5, 2021

The purpose of this article is to provide assistance if your upgrade to OpenAM 12.0.0 and 12.0.1 hangs in DirectoryContentUpgrader when configuration and CTS stores are external (either running on the same or separate OpenDJ instances).


1 reader recommends this article

Archived

This article has been archived and is no longer maintained by ForgeRock.

Symptoms

An error similar to the following is shown in the amUpgrade log when the upgrade hangs:

amUpgrade:08/10/2015 01:18:25:955 PM UTC: Thread[http-bio-38080-exec-59,5,main] ERROR: An error occurred while trying to get a connection org.forgerock.opendj.ldap.ConnectionException: Connect Error: No operational connection factories available at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:210) at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:172) at org.forgerock.opendj.ldap.ErrorResultException.newErrorResult(ErrorResultException.java:142) at org.forgerock.opendj.ldap.AbstractLoadBalancingAlgorithm.getMonitoredConnectionFactory(AbstractLoadBalancingAlgorithm.java:389) at org.forgerock.opendj.ldap.AbstractLoadBalancingAlgorithm.access$100(AbstractLoadBalancingAlgorithm.java:55) at org.forgerock.opendj.ldap.AbstractLoadBalancingAlgorithm$MonitoredConnectionFactory.getConnection(AbstractLoadBalancingAlgorithm.java:84) at org.forgerock.opendj.ldap.LoadBalancer.getConnection(LoadBalancer.java:53) at org.forgerock.openam.upgrade.DirectoryContentUpgrader.upgrade(DirectoryContentUpgrader.java:210) at org.forgerock.openam.upgrade.steps.UpgradeDirectoryContentStep.perform(UpgradeDirectoryContentStep.java:72) at org.forgerock.openam.upgrade.UpgradeServices.upgrade(UpgradeServices.java:186) at com.sun.identity.config.upgrade.Upgrade.doUpgrade(Upgrade.java:79) ...

Correspondingly, you may see BLOCKED or WAITING threads relating to DirectoryContentUpgrader when the upgrade hangs, similar to the following example stack traces:

  • BLOCKED: Thread 17573: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=834 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=994 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1303 (Interpreted frame) - com.forgerock.opendj.util.AsynchronousFutureResult$Sync.innerGet() @bci=2, line=144 (Interpreted frame) - com.forgerock.opendj.util.AsynchronousFutureResult.get() @bci=4, line=267 (Interpreted frame) - org.forgerock.opendj.ldap.CachedConnectionPool.getConnection() @bci=5, line=772 (Interpreted frame) - org.forgerock.opendj.ldap.AbstractLoadBalancingAlgorithm$MonitoredConnectionFactory.getConnection() @bci=4, line=79 (Interpreted frame) - org.forgerock.opendj.ldap.LoadBalancer.getConnection() @bci=9, line=53 (Interpreted frame) - org.forgerock.openam.upgrade.DirectoryContentUpgrader.upgrade() @bci=6, line=210 (Interpreted frame) - org.forgerock.openam.upgrade.steps.UpgradeDirectoryContentStep.perform() @bci=4, line=72 (Interpreted frame) - org.forgerock.openam.upgrade.UpgradeServices.upgrade(com.iplanet.sso.SSOToken, boolean) @bci=94, line=186 (Interpreted frame) - com.sun.identity.config.upgrade.Upgrade.doUpgrade() @bci=12, line=79 (Interpreted frame) ...
  • WAITING: http-bio-8080-exec-3" daemon prio=3 tid=0x0000000105e37800 nid=0xae waiting on condition [0xffffffff6bdfa000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007b2cd1e20> (a com.forgerock.opendj.util.AsynchronousFutureResult$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at com.forgerock.opendj.util.AsynchronousFutureResult$Sync.innerGet(AsynchronousFutureResult.java:144) at com.forgerock.opendj.util.AsynchronousFutureResult.get(AsynchronousFutureResult.java:267) at org.forgerock.opendj.ldap.CachedConnectionPool.getConnection(CachedConnectionPool.java:772) at org.forgerock.opendj.ldap.AbstractLoadBalancingAlgorithm$MonitoredConnectionFactory.getConnection(AbstractLoadBalancingAlgorithm.java:79) at org.forgerock.opendj.ldap.LoadBalancer.getConnection(LoadBalancer.java:53) at org.forgerock.openam.upgrade.DirectoryContentUpgrader.upgrade(DirectoryContentUpgrader.java:210) at org.forgerock.openam.upgrade.steps.UpgradeDirectoryContentStep.perform(UpgradeDirectoryContentStep.java:72) at org.forgerock.openam.upgrade.UpgradeServices.upgrade(UpgradeServices.java:186) at com.sun.identity.config.upgrade.Upgrade.doUpgrade(Upgrade.java:79) ...
Note

If you need help capturing stack traces, see How do I collect JVM data for troubleshooting AM (All versions)? for further information.

Recent Changes

N/A

Causes

The default connection pool sizing of 10 is too small due to the way the upgrade process creates connection factories. Due to this sizing issue, the upgrade process hangs at this point.

Solution

This issue can be resolved by upgrading to OpenAM 12.0.2 or later; you can download this from BackStage.

Workaround

Alternatively, this issue can be resolved by temporarily overriding your maximum connection pool size values in your web application container and then performing the upgrade again.

You can override your maximum connection pool size values by setting the following JVM property:

-Dmax_conn_pool=24

Example using Apache Tomcat™ web container

You would override your maximum connection pool size values by specifying CATALINA_OPTS settings in the setenv.sh file (typically located in the /tomcat/bin/ directory). If this file doesn't exist, you should create it in the same directory as the catalina.sh file (also typically located in the /tomcat/bin/directory).

To override your maximum connection pool size values:

  1. Add the following line to the setenv.sh file: export CATALINA_OPTS="-Dmax_conn_pool=24"
  2. Restart the web container.

See Also

Upgrade to OpenAM 12.0.0 or 12.0.1 fails with Entry Already Exists exception when CTS store is external with non-default suffix

Upgrade to AM 5.x or 6.x fails when anonymous access is disabled in DS

Best practice for upgrading to OpenAM 12.x

Related Training

N/A

Related Issue Tracker IDs

OPENAM-6455 (ConnectionCount logic does not produce a sensible ConnectionFactory max pool size for some scenarios)

OPENAM-6456 (Upgrading to OpenAM 12 hangs in DirectoryContentUpgrader)


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.