Solutions
ForgeRock Identity Platform
Does not apply to Identity Cloud

High thread count seen in AM (All versions) with many RUNNABLE I/O dispatcher threads

Last updated Dec 20, 2021

The purpose of this article is to provide assistance if you see increasing numbers of threads being created by AM but not released. You will notice there are lots of I/O dispatcher threads with a RUNNABLE state. You may also experience symptoms such as AM crashing, high memory usage and so on.


Symptoms

You will observe that AM is creating increasing numbers of threads but not releasing them. You may also experience symptoms such as AM crashing, high memory usage and so on.

Repeated stack traces similar to the following are shown when you collect a thread dump in AM:"I/O dispatcher <number>" #1234 prio=5 os_prio=0 tid=0x00007fba600c2000 nid=0x225 runnable [0x00007fbad10c1000]   java.lang.Thread.State: RUNNABLE       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)       at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)       - locked <0x00000006f7400358> (a sun.nio.ch.Util$3)       - locked <0x00000006f7400230> (a sun.nio.ch.EPollSelectorImpl)       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)       at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)       at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)       at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)       at java.lang.Thread.run(Thread.java:748)    Locked ownable synchronizers:       - None

You can collect a thread dump as described in How do I collect JVM data for troubleshooting AM (All versions)?

Recent Changes

Implemented or updated custom code.

Causes

Large numbers of I/O dispatcher threads are typically seen in applications that do the following:

  • Use httpClient or AsyncHttpClient for HTTP requests.
  • Open outbound connections but do not close them properly.

AM does not do this by default, so these I/O dispatcher thread leaks are usually the result of custom code or applications that make outbound HTTP connections without closing them.

Solution

This issue can be resolved by scanning through all your custom code, scripts and applications to look for instances of httpClient or AsyncHttpClient making outbound HTTP requests without closing them. When a request is made using httpClient (HttpClientHandler) or Apache AsyncHttpClient, a worker thread (with a name such as I/O dispatcher <id> or pool-<id>-thread-<id>) may be involved to handle the asynchronous request/response. 

Once identified, you will then need to modify your code to ensure all I/O resources and clients are closed appropriately, or use singleton copies of HttpHandler classes.

Note

Creating and fixing custom code is outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services.

See Also

How do I find which thread is consuming CPU in a Java process in AM (All versions)?

How do I collect data for troubleshooting high CPU utilization on AM (All versions) servers?

Package org.forgerock.http.handler

Class HttpClientHandler

Related Training

N/A

Related Issue Tracker IDs

N/A


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.