How do I collect data for troubleshooting high CPU utilization or Out of Memory errors on IG (All versions) servers?
The purpose of this article is to provide troubleshooting guidance if you are experiencing consistently high CPU utilization on the IG server or seeing Out of Memory errors. If CPU utilization is in the range 70-80% under load, you should definitely investigate what is causing it as performance is likely to be impacted.
1 reader recommends this article
Troubleshooting high CPU usage or Out of Memory errors
Both high CPU utilization and Out of Memory errors are usually experienced in conjunction with other symptoms such as servers becoming unresponsive and poor performance.
Note
On Linux®, Unix® and Solaris® systems, there is a script you can run to capture the stack traces and process information as detailed in How do I use the msnapshots script to capture information for troubleshooting IG (All versions)? This script only applies if IG is running in a web container.
There are a number of different types of data you should collect to troubleshoot high CPU utilization or Out of Memory errors; you should collect the following data and submit it to us when you raise the ticket to enable us to help you more quickly:
- Debug logs
- Configuration directory
- JVM stack traces and current options
- Process / thread information
- Garbage Collector (GC) logs
In addition, please include any information about other symptoms you are seeing at the same time as the issue and any changes you have made to your system that may have contributed to the problem.
Note
It is important that you collect all the data at the point you are experiencing the issue so that the timestamps of the data collected and the issue correspond. Please notify us of the exact timestamp when high CPU / Out of Memory errors was noticed so we can focus our attention on the relevant information.
Debug logs
Debug level debug logs provide more context for diagnosing errors but can generate large files. You should collect logs when you are experiencing an issue as detailed in How do I generate more detailed debug logs to diagnose an issue in IG (All versions)?
Configuration directory
The configuration directory ($HOME/.openig/config/) contains all the configuration settings for an IG server instance and is useful for checking settings that may play a role in the issue.
JVM stack traces and current options
JVM stack traces must be collected before killing the affected process or restarting the server; otherwise the information relating to the affected process is lost forever, which may make it impossible to identify the root cause. In addition to collecting stack traces, you should also provide details of the current JVM settings as they can provide an opportunity for tuning, which can reduce CPU usage or Out of Memory errors.
Collecting stack traces and finding current JVM settings is described in How do I collect JVM data for troubleshooting IG (All versions)? This information only applies if IG is running in a web container.
Process / thread information
Detailed process / thread information is useful for identifying the exact thread within a process that is experiencing the high CPU. You should collect this data at the same time as the stack trace so we can look for the identified thread in the stack trace for more information. This information only applies if IG is running in a web container.
- On Unix and Linux systems, you can use the top command to output process information for all threads to a file: $ top -b -n 1 -H -p [pid] > top-output.txtreplacing [pid] with the process ID of the affected process.
- On Solaris, you can use the prstat command to capture this information: $ prstat -L -p [pid]replacing [pid] with the process ID of the affected process.
- On Microsoft® Windows®, you will need to use the Process Monitor.
See How do I find which thread is consuming CPU in a Java process in IG (All versions)? for further information on identifying the thread that is causing high CPU usage.
GC logs
GC logs also provide useful information about tuning issues that may be contributing to the high CPU usage or Out of Memory errors. Enabling GC logging is described in How do I enable Garbage Collector (GC) Logging for IG (All versions)? This information only applies if IG is running in a web container.
Once you have collected your GC logs, you should refer to Best practice for JVM Tuning with G1 GC or Best practice for JVM Tuning with CMS GC for advice on resolving common issues identified in the logs.
See Also
Where can I find useful logs for troubleshooting ForgeRock products?
Related Training
N/A
Related Issue Tracker IDs
N/A