Solutions
ForgeRock Identity Platform
Does not apply to Identity Cloud

Memory leak messages when shutting down Apache Tomcat web container running AM (All versions)

Last updated Jun 29, 2021

The purpose of this article is to provide assistance if you see memory leak messages when shutting down the Apache Tomcat™ web container running AM. You will see messages such as "WARNING: The web application [AM] appears to have started a thread named [ ] but has failed to stop it. This is very likely to create a memory leak." and "SEVERE: The web application [AM] created a ThreadLocal with key of type [ ] (value [ ]) and a value of type [ ] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak."


2 readers recommend this article

Symptoms

Messages similar to the following are shown in catalina.out when shutting down the Tomcat web container: 28-Jun-2021 16:28:03.418 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [AM] appears to have started a thread named [OpenDJ LDAP SDK Client Worker(1)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:   sun.misc.Unsafe.park(Native Method)    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)    java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)    java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)    java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)    java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)    java.lang.Thread.run(Thread.java:745) 28-Jun-2021 16:28:03.419 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [AM] appears to have started a thread named [OpenDJ LDAP SDK Default Scheduler] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: ...

Or:15-Mar-2021 18:19:43 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads WARNING: The web application [AM] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method)  java.lang.ref.ReferenceQueue.remove(Unknown Source)  java.lang.ref.ReferenceQueue.remove(Unknown Source)  com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114) ... 15-Mar-2021 18:19:43 PM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks SEVERE: The web application [sso] created a ThreadLocal with key of type [com.sun.identity.common.LDAPConnectionPool$ThreadLocalConnection$2] (value [com.sun.identity.common.LDAPConnectionPool$ThreadLocalConnection$2@23233749]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ...

Or:15-Mar-2021 18:19:43.888 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [AM] appears to have started a thread named [Time Thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method)  java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)  java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) ... 15-Mar-2021 18:19:44.914 INFO [Directory Server Shutdown Hook] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.opends.server.extensions.JMXAlertHandler]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.  java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.opends.server.extensions.JMXAlertHandler]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. ...

Note

These messages can also occur if you repeatedly redeploy the AM war whilst Tomcat is running; AM is not intended to be un-deployed/deployed without a container restart.

Recent Changes

N/A

Causes

You can find more information on these memory leak messages at Tomcat Wiki - MemoryLeakProtection.

Solution

These messages can be safely ignored providing Tomcat is not prevented from shutting down. As they occur during shutdown, any memory leaks will cease once the process has ended.

If Tomcat fails to shut down properly, you can forcibly end the process using kill -9 as follows:

  1. Identify the process ID using the following command: $ ps –ef |grep tomcatExample response, where process ID is 2240: forgero+ 2240   1  3 10:34 pts/1    00:01:31 /usr/lib/jvm/java-8-oracle/bin/java -Djava.util.logging.config.file=/opt/tomcat/am7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -Xss256k -XX:+UseParallelGC -XX:MaxGCPauseMillis=1500 -XX:GCTimeRatio=9 -server -XX:+DisableExplicitGC -Djava.endorsed.dirs=/opt/tomcat/am7/endorsed -classpath /opt/tomcat/am7/bin/bootstrap.jar:/opt/tomcat/am7/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat/am7 -Dcatalina.home=/opt/tomcat/am7 -Djava.io.tmpdir=/opt/tomcat/am7/temp org.apache.catalina.startup.Bootstrap start forgero+  2966  2193  0 11:14 pts/1    00:00:00 grep --color=auto tomcat
  2. End the process using the following command and the identified process ID, for example: $ kill -9 2240
  3. Repeat step 2 if more than one process is running for the AM instance you are trying to shut down (this can happen if you started and stopped the instance again).

See Also

AM (All versions) fails to start due to SEVERE: ContainerBase.addChild: start: error on Apache Tomcat

Tomcat Wiki - MemoryLeakProtection

Related Training

N/A

Related Issue Tracker IDs

OPENAM-13962 (Errors during shutdown of AM)

OPENAM-3133 (SEVERE: The web application [/openam] created a ThreadLocal...but failed to remove it)


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.