OpenAM 13.0 hangs when creating or updating policies and policy sets with BLOCKED and WAITING threads
The purpose of this article is to provide assistance if OpenAM 13.0 hangs when creating or updating policies and policy sets, and you see BLOCKED or WAITING threads. You may also encounter these symptoms in other policy related scenarios, including exporting policies using the ssoadm list-xacml command, accessing UMA resources and upgrading with a large number of policies.
1 reader recommends this article
Archived
This article has been archived and is no longer maintained by ForgeRock.
Symptoms
OpenAM intermittently hangs when creating or updating policies and policy sets, exporting policies using the ssoadm list-xacml command or accessing UMA resources. Once it hangs, restarting the web application container in which OpenAM runs will temporarily restore functionality.
You will see either BLOCKED or WAITING threads in a JStack when OpenAM hangs, for example:
http-/0.0.0.0:8080-14" #109 daemon prio=5 os_prio=0 tid=0x0000000022a0d800 nid=0xfc0 in Object.wait() [0x000000002617a000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.forgerock.util.promise.PromiseImpl.await(PromiseImpl.java:572) - locked <0x00000000c7a5ec00> (a org.forgerock.util.promise.PromiseImpl) at org.forgerock.util.promise.PromiseImpl.getOrThrow(PromiseImpl.java:143) at org.forgerock.opendj.ldap.CachedConnectionPool.getConnection(CachedConnectionPool.java:789) at com.sun.identity.sm.ldap.SMDataLayer.getConnection(SMDataLayer.java:107) at com.sun.identity.sm.ldap.SMSLdapObject.getConnection(SMSLdapObject.java:574) at com.sun.identity.sm.ldap.SMSLdapObject.search(SMSLdapObject.java:591) at com.sun.identity.sm.SMSEntry.search(SMSEntry.java:1069) at com.sun.identity.entitlement.opensso.DataStore.searchPrivileges(DataStore.java:902) at com.sun.identity.entitlement.opensso.DataStore.search(DataStore.java:822) at com.sun.identity.entitlement.opensso.OpenSSOIndexStore$SearchTask.run(OpenSSOIndexStore.java:96) ... at com.sun.identity.entitlement.ApplicationPrivilegeManager.getInstance(ApplicationPrivilegeManager.java:188) Thread 2931: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Interpreted frame) - org.forgerock.opendj.ldif.ConnectionEntryReader.getNextResponse() @bci=21, line=398 (Interpreted frame) - org.forgerock.opendj.ldif.ConnectionEntryReader.hasNext() @bci=1, line=231 (Interpreted frame) - com.sun.identity.sm.ldap.SMSLdapObject.searchObjects(com.iplanet.sso.SSOToken, java.lang.String, java.lang.String, int, int, boolean, boolean, org.forgerock.opendj.ldap.Connection) @bci=81, line=702 (Interpreted frame)Upgrade process
This issue can also cause the upgrade process to hang with one of the following errors in the amUpgrade debug log, in addition to BLOCKED threads:
ERROR: Error occured while upgrading OpenAM org.forgerock.openam.upgrade.UpgradeException: Failed to gather policies for application iPlanetAMWebAgentService at org.forgerock.openam.upgrade.steps.policy.UpgradeResourceTypeStep.upgradePrivileges(UpgradeResourceTypeStep.java:416) at org.forgerock.openam.upgrade.steps.policy.UpgradeResourceTypeStep.perform(UpgradeResourceTypeStep.java:249) at org.forgerock.openam.upgrade.UpgradeServices.upgrade(UpgradeServices.java:151) at com.sun.identity.config.upgrade.Upgrade.doUpgrade(Upgrade.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)ERROR: Error occured while upgrading OpenAM org.forgerock.openam.upgrade.UpgradeException: Application cloning failed at org.forgerock.openam.upgrade.steps.RemoveReferralsStep.instateReferredApplications(RemoveReferralsStep.java:204) at org.forgerock.openam.upgrade.steps.RemoveReferralsStep.perform(RemoveReferralsStep.java:195) at org.forgerock.openam.upgrade.UpgradeServices.upgrade(UpgradeServices.java:151) at com.sun.identity.config.upgrade.Upgrade.doUpgrade(Upgrade.java:83)Recent Changes
Upgraded to, or installed OpenAM 13.0.
Causes
Connections are being opened but not closed and returned to the connection pool, which eventually causes OpenAM to hang. During the upgrade process, this can happen when the upgrade reaches the policy section and is migrating policies or removing policy referrals.
Solution
This issue can be resolved by upgrading to OpenAM 13.5 or later; you can download this version from BackStage.
See Also
Best practice for migrating policies when upgrading to OpenAM 12.x or 13.x
Upgrade to OpenAM 13.x fails with Failed to modify privilege! message when migrating policies
How do I export and import policies in AM (All versions)?
Related Training
N/A
Related Issue Tracker IDs
OPENAM-8531 (Connection leak in policy codebase)
OPENAM-8459 (LDAPAuthUtils should set operations timeout in seconds)
OPENAM-10116 (Search timeout for LDAP filter condition should be in seconds)