How do I collect JVM data for troubleshooting IG 6.x and 7.x?

Last updated Feb 23, 2023

The purpose of this article is to provide information on collecting JVM data for troubleshooting IG when it is running in a web container. JVM data (such as stack traces, heap histograms and heap dumps) can be very useful for troubleshooting issues such as a hung IG server, Out of Memory errors and suspected memory leaks. This data must be collected before killing the affected process or restarting the server; otherwise the information relating to the affected process is lost forever, which may make it impossible to identify the root cause.

Note

This article does not apply to IG 2023.2 and later because you cannot run IG in a web container in these later versions.

The following utilities are all JDK utilities and are not included with the JRE; you must have the JDK installed in your production environments in order to collect this data. Additionally, you must be the owner of the identified process to be able to collect this data.

In Java 11 (11.0.5 and later), you can use the jhsdb tool to capture stack traces, heap histograms and heap dumps instead of using jstack and jmap.

Collecting JVM data

To collect JVM data, you must first determine the process ID of the hung web container instance using the following command:

$ jps -v

You can then collect the following types of data:

Stack Traces
Heap Histograms
Heap Dumps
Current JVM settings

Stack Traces

Stack traces allow us to see what is happening with JVM threads and are sometimes referred to as thread dumps. You should collect several different stack traces (minimum of three) at roughly one minute intervals to enable us to compare them. It is also beneficial to collect top outputs at the same time so we can monitor processes and system resource usage.

Finally, it is recommended that you also identify the threads associated with the process ID for troubleshooting high CPU as detailed in How do I find which thread is consuming CPU in a Java process in IG 6.x and 7.x?

Stack traces

You can use the following commands to collect stack traces:

You can collect long listing stack traces using the jstack command: $ jstack -l [pid] > [outputfile.txt]replacing [pid] with the process ID of the hung container instance and [outputfile.txt] with the output file for the stack trace. In Java 8, you can include the -F option to force a stack dump if needed.
You can use the kill -3 command (Unix® and Linux® systems only), which consistently produces better results:$ kill -3 [pid]replacing [pid] with the process ID of the affected server.

The kill -3 command outputs the stack trace to the log file where the standard output is directed (typically the error log file). The kill -3 command does not stop the running process; it will continue to run after outputting the stack trace.

You can use the Java Thread Dump Analyzer to analyze your stack traces. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

Top output

You can use the following commands to collect top outputs:

On Unix® and Linux® systems, you can use the top command: $ top -n 1 -H -p [pid]replacing [pid] with the process ID of the affected process.
On Solaris®, you can use the prstat command: $ prstat -L -p [pid]replacing [pid] with the process ID of the affected process.
On Microsoft® Windows®, you can use the Process Monitor.

Heap Histograms

Heap histograms enable us to visualize the objects in the JVM heap - we are only interested in live objects. You can collect heap histograms using the jmap command:

$ jmap -histo:live [pid]

replacing [pid] with the process ID of the hung container instance.

Caution

The jmap utility must come from the same version of JVM that the container is running.

Heap Dumps

Heap dumps allow us to see what is happening with memory at the point the heap dump is taken, which can help us to identify high memory usage issues or memory leaks. You should collect heap dumps as close in time to the stack traces as possible to enable us to compare the contents. You can collect heap dumps using the jmap command:

$ jmap -dump:file=[target file] [pid]

replacing [target file] with the output file for the heap dump and [pid] with the process ID of the hung container instance. In Java 8, you can include the -F option to force a heap dump if needed.

Current JVM settings

You can find the current JVM settings for the server using one of the following commands (the jinfo command is the preferred way):

jinfo: $ jinfo [pid]replacing [pid] with the required process ID.
jps: $ jps -v

For Unix® and Linux® systems where IG is running on an Apache Tomcat™ web container, you can also use:

$ ps -ef | grep tomcat