High thread count seen in AM (All versions) with many RUNNABLE I/O dispatcher threads
The purpose of this article is to provide assistance if you see increasing numbers of threads being created by AM but not released. You will notice there are lots of I/O dispatcher threads with a RUNNABLE state. You may also experience symptoms such as AM crashing, high memory usage and so on.
Symptoms
Repeated stack traces similar to the following are shown when you collect a thread dump in AM:"I/O dispatcher <number>" #1234 prio=5 os_prio=0 tid=0x00007fba600c2000 nid=0x225 runnable [0x00007fbad10c1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000006f7400358> (a sun.nio.ch.Util$3) - locked <0x00000006f7400230> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None
You can collect a thread dump as described in How do I collect JVM data for troubleshooting AM?
Recent Changes
Implemented or updated custom code.
Causes
Large numbers of I/O dispatcher
threads are typically seen in applications that do the following:
- Use httpClient or AsyncHttpClient for HTTP requests.
- Open outbound connections but do not close them properly.
AM does not do this by default, so these I/O dispatcher
thread leaks are usually the result of custom code or applications that make outbound HTTP connections without closing them.
Solution
This issue can be resolved by scanning through all your custom code, scripts and applications to look for instances of httpClient or AsyncHttpClient making outbound HTTP requests without closing them. When a request is made using httpClient (HttpClientHandler) or Apache AsyncHttpClient, a worker thread (with a name such as I/O dispatcher <id>
or pool-<id>-thread-<id>
) may be involved to handle the asynchronous request/response.
Once identified, you will then need to modify your code to ensure all I/O resources and clients are closed appropriately, or use singleton copies of HttpHandler classes.
Note
Creating and fixing custom code is outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services.
See Also
How do I find which thread is consuming CPU in a Java process in AM?
How do I collect data for troubleshooting high CPU utilization on AM (All versions) servers?
Package org.forgerock.http.handler
Related Training
N/A
Related Issue Tracker IDs
N/A