You can run the Support Extract tool to make it easier to capture the data listed below (along with other valuable troubleshooting data) as detailed in supportextract — extract support data (DS 6 and later) or How do I use the Support Extract tool in DS (All versions) to capture troubleshooting data?
The following utilities are all JDK utilities and are not included with the JRE; you must have the JDK installed in your production environments in order to collect this data. Additionally, you must be the owner of the identified process to be able to collect this data.
To collect JVM data, you must first determine the process ID of the affected server process using the following command:$ jps -v
You can then collect the following types of data:
Stack traces allow us to see what is happening with JVM threads and are sometimes referred to as thread dumps. You should collect several stack traces (minimum of three) at roughly one minute intervals to enable us to compare them. It is also beneficial to collect top outputs at roughly the same times so we can monitor processes and system resource usage.
Finally, it is recommended that you also identify the threads associated with the process ID for troubleshooting high CPU as detailed in How do I find which thread is consuming CPU in a Java process in DS (All versions)?
You can use the following commands to collect stack traces:
- You can collect long listing stack traces using the jstack command:$ jstack -F -l [pid] > [outputfile.txt]replacing [pid] with the process ID of the affected server and [outputfile.txt] with the output file for the stack trace.
- You can use the kill -3 command (Unix® and Linux® systems only), which consistently produces better results:$ kill -3 [pid]replacing [pid] with the process ID of the affected server.
The kill -3 command outputs the stack trace to the log file where the standard output is directed (typically the error log file). The kill -3 command does not stop the running process; it will continue to run after outputting the stack trace.
You can use the Java Thread Dump Analyzer to analyze your stack traces. This is a third-party website that we suggest can be used for analysis but is not supported by ForgeRock.
You can use the following commands to collect top outputs:
- On Unix® and Linux® systems, you can use the top command: $ top -n 1 -H -p [pid]replacing [pid] with the process ID of the affected process.
- On Solaris®, you can use the prstat command: $ prstat -L -p [pid]replacing [pid] with the process ID of the affected process.
- On Microsoft® Windows®, you can use the Process Monitor.
Heap histograms enable us to visualize the objects in the JVM heap - we are only interested in live objects. You can collect heap histograms using the jmap command:$ jmap -histo:live [pid]
replacing [pid] with the process ID of the hung container instance.
The jmap utility must come from the same version of JVM that the container is running.
Heap dumps allow us to see what is happening with memory at the point the heap dump is taken, which can help us to identify high memory usage issues or memory leaks. You should collect heap dumps as close in time to the stack traces as possible to enable us to compare the contents. You can collect heap dumps using the jmap command:$ jmap -F -dump:file=[target file] [pid]
replacing [target file] with the output file for the heap dump and [pid] with the process ID of the affected server.
- jinfo: $ jinfo [pid]replacing [pid] with the required process ID.
- jps: $ jps -v
For Unix® and Linux® systems where AM is running on an Apache Tomcat™ web container, you can also use:$ ps -ef | grep tomcat