How To
ForgeRock Identity Platform
Does not apply to Identity Cloud

How do I collect JVM data for troubleshooting DS (All versions)?

Last updated Sep 2, 2021

The purpose of this article is to provide information on collecting JVM data for troubleshooting DS. JVM data (such as stack traces, heap histograms and heap dumps) can be very useful for troubleshooting DS issues such as slow response times, consistently high CPU usage, Out of Memory errors and suspected memory leaks. This data must be collected before killing the affected process or restarting the server; otherwise the information relating to the affected process is lost forever, which may make it impossible to identify the root cause.


1 reader recommends this article

Collecting JVM data

Note

You can run the Support Extract tool to make it easier to capture the data listed below (along with other valuable troubleshooting data) as detailed in supportextract — extract support data (DS 6 and later) or How do I use the Support Extract tool in DS (All versions) to capture troubleshooting data? 

The following utilities are all JDK utilities and are not included with the JRE; you must have the JDK installed in your production environments in order to collect this data. Additionally, you must be the owner of the identified process to be able to collect this data.

To collect JVM data, you must first determine the process ID of the affected server process using the following command:

$ jps -v

You can then collect the following types of data:

Stack Traces

Stack traces allow us to see what is happening with JVM threads and are sometimes referred to as thread dumps. You should collect several stack traces (minimum of three) at roughly one minute intervals to enable us to compare them. It is also beneficial to collect top outputs at roughly the same times so we can monitor processes and system resource usage.

Finally, it is recommended that you also identify the threads associated with the process ID for troubleshooting high CPU as detailed in How do I find which thread is consuming CPU in a Java process in DS (All versions)?

Stack traces

You can use the following commands to collect stack traces:

  • You can collect long listing stack traces using the jstack command:$ jstack -F -l [pid] > [outputfile.txt]replacing [pid] with the process ID of the affected server and [outputfile.txt] with the output file for the stack trace.
  • You can use the kill -3 command (Unix® and Linux® systems only), which consistently produces better results:$ kill -3 [pid]replacing [pid] with the process ID of the affected server.

The kill -3 command outputs the stack trace to the log file where the standard output is directed (typically the error log file). The kill -3 command does not stop the running process; it will continue to run after outputting the stack trace.

You can use the Java Thread Dump Analyzer to analyze your stack traces. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.

Top output 

You can use the following commands to collect top outputs:

  • On Unix® and Linux® systems, you can use the top command: $ top -n 1 -H -p [pid]replacing [pid] with the process ID of the affected process.
  • On Solaris®, you can use the prstat command: $ prstat -L -p [pid]replacing [pid] with the process ID of the affected process.
  • On Microsoft® Windows®, you can use the Process Monitor.

Heap Histograms

Heap histograms enable us to visualize the objects in the JVM heap - we are only interested in live objects. You can collect heap histograms using the jmap command:

$ jmap -histo:live [pid]

replacing [pid] with the process ID of the hung container instance.

Caution

The jmap utility must come from the same version of JVM that the container is running.

Heap Dumps

Heap dumps allow us to see what is happening with memory at the point the heap dump is taken, which can help us to identify high memory usage issues or memory leaks. You should collect heap dumps as close in time to the stack traces as possible to enable us to compare the contents. You can collect heap dumps using the jmap command:

$ jmap -F -dump:file=[target file] [pid]

replacing [target file] with the output file for the heap dump and [pid] with the process ID of the affected server.

Current JVM settings

You can find the current JVM settings for the server using one of the following commands (the jinfo command is the preferred way):

  • jinfo: $ jinfo [pid]replacing [pid] with the required process ID.
  • jps: $ jps -v

For Unix® and Linux® systems where AM is running on an Apache Tomcat™ web container, you can also use: 

$ ps -ef | grep tomcat

See Also

How do I use the msnapshots script to capture information for troubleshooting DS (All versions)?

How do I collect data for troubleshooting high CPU utilization on DS (All versions) servers?

How do I enable Garbage Collector (GC) Logging for DS (All versions)?

How do I tune DS (All versions) process sizes: JVM heap and database cache?

Troubleshooting DS

Related Training

N/A

Related Issue Tracker IDs

N/A


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.