High thread count seen in AM (All versions) with many RUNNABLE I/O dispatcher threads

Last updated Jan 16, 2023

The purpose of this article is to provide assistance if you see increasing numbers of threads being created by AM but not released. You will notice there are lots of I/O dispatcher threads with a RUNNABLE state. You may also experience symptoms such as AM crashing, high memory usage and so on.

Symptoms

You will observe that AM is creating increasing numbers of threads but not releasing them. You may also experience symptoms such as AM crashing, high memory usage and so on.

Repeated stack traces similar to the following are shown when you collect a thread dump in AM:"I/O dispatcher <number>" #1234 prio=5 os_prio=0 tid=0x00007fba600c2000 nid=0x225 runnable [0x00007fbad10c1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000006f7400358> (a sun.nio.ch.Util$3) - locked <0x00000006f7400230> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None

You can collect a thread dump as described in How do I collect JVM data for troubleshooting AM?

Recent Changes

Implemented or updated custom code.

Causes

Large numbers of I/O dispatcher threads are typically seen in applications that do the following:

Use httpClient or AsyncHttpClient for HTTP requests.
Open outbound connections but do not close them properly.

AM does not do this by default, so these I/O dispatcher thread leaks are usually the result of custom code or applications that make outbound HTTP connections without closing them.

Solution

This issue can be resolved by scanning through all your custom code, scripts and applications to look for instances of httpClient or AsyncHttpClient making outbound HTTP requests without closing them. When a request is made using httpClient (HttpClientHandler) or Apache AsyncHttpClient, a worker thread (with a name such as I/O dispatcher <id> or pool-<id>-thread-<id>) may be involved to handle the asynchronous request/response.

Once identified, you will then need to modify your code to ensure all I/O resources and clients are closed appropriately, or use singleton copies of HttpHandler classes.

Note

Creating and fixing custom code is outside the scope of ForgeRock support; if you want more tailored advice, consider engaging Deployment Support Services.

Related Training

N/A

Related Issue Tracker IDs

N/A